The big_query() function

The big_query() function allows programmers more control than the simpler query() function on how data is retrieved from the database server, as it allows fetching the data rows on demand. This is especially useful when you wish to do client-side computations on the fly on big datasets, that would require too much memory to be completely fetched and then processed.

The function's signature is object(Sql.sql_result) big_query(string sql)

The returned object is a handle to the results dataset. It offers methods allowing you to retrieve rows and get informations on the dataset itself.

int num_rows(): returns the total number of rows in the result object. Some drivers (i.e. Sybase) might not provide this functionality, and thus the only way to know how many rows there are is by explicitly querying the server (see example below).
int num_fields(): returns the number of columns for the result object. This function is usually meant for development purposes only, you shouldn't need it on production systems.
int eof(): returns true if all rows in the result object have been fetched.
array(mapping(string:mixed)) fetch_fields(): retrieves descriptions for the columns in the results set. The mappings in the returned array (one for each column) have some default fields, but they change in different drivers. See the example below to discover what fields your driver of choice provides. This function is usually used for development purposes only. You should rarely need it on production systems. Also, notice that the returned results will correspond to the server's idea of the fields, which might be different from the actual declaration.
void seek(int skip): This method allows to skip fetching some rows (the skip argument must be a nonnegative integer).
int|array(string|int) fetch_row(): The most important function of all, this one allows you to fetch a row of data. There is one element of the array for each column, and the columns are ordered as returned by fetch_fields() and as specified in the SQL query. If 0 is returned instead, it means that there are no more rows to retrieve. An integer 0 is returned for (SQL) NULL values, while all types of stored data are returned as strings. It's up to the user to do the adequate type casts where appropriate. Type information can usually be retrieved with the fetch_fields() function.

Note!

There are some restrictions on how data are retrieved with some drivers. Please check the drivers-specific section for more detailed information.

Print the name and background for all the countries in Europe.

object(Sql.sql) db=Sql.sql("mysql://user:password@localhost/sample");
object(Sql.sql_result) result=db->big_query(
       "select ids.name, countries.background "
       "from ids,countries,areas "
       "where areas.name='Europe' and countries.map_refs=areas.id and "
       "ids.code=countries.country");
array(string) row;
while (row=result->fetch_row())
{
      //row[0] is the country name, row[1] is the background info
      write("---"+row[0]+"\n");
      write(row[1]+"\n");
}

Now let's try writing a simple pikescript handling a multi-page table without resorting to the LIMIT SQL clause (see ../data_extract/limiting). The main purpose of this example is showing the usage of num_rows and seek functions, so despite being a complete example, it's a bit stretched (in real-world, this is one of the cases where the Roxen WebServer caching capabilities come handy). Also, it doesn't output formally valid HTML, and it doesn't handle exceptions. We'll show the 'ids' table contents, with ten entries per page and links to the other pages.

#define DBHOST "mysql://user:password@localhost/sample"
#define QUERY "select name, code from ids order by name"
#define ENTRIES_PER_PAGE 10
#define SEEK_IS_BROKEN

string parse (object id)
{
  string toreturn;
  object(Sql.sql) db;
  int number_of_entries, number_of_pages, page, j;
  object(Sql.sql_result) result;
  array(string) row;
  
  page=(int)(id->variables->page);
  toreturn="<table border=1>\n";
  db=Sql.sql(DBHOST);                                 //connect
  result=db->big_query(QUERY);                        //query
  number_of_entries=result->num_rows();               //get the number of rows
#ifdef SEEK_IS_BROKEN
  //it looks like mysql's implementation of seek() is broken, probably at
  //the mysql level in my version (3.22.29). I'll do a loop to emulate seek
  for (j=0;j<ENTRIES_PER_PAGE*page;j++)
    result->fetch_row();
#else
  result->seek(ENTRIES_PER_PAGE*page);                //skip unneeded results
#endif

  for(j=0; j<10; j++) {                //at most 10 results
    row=result->fetch_row();           //fetch the row              
    if (!row)                          //no more data?
      break;                             //exit
    toreturn += "<tr><td>"+row[0]+"</td><td>"+row[1]+"</td></tr>\n";
  }
                                       //now the links section
  number_of_pages=number_of_entries/ENTRIES_PER_PAGE;
  if (number_of_entries%ENTRIES_PER_PAGE)
    number_of_pages++;                 //there might be an incomplete page
  toreturn+="<tr><td colspan=2>";
  for (j=0;j<number_of_pages;j++)
  {
    toreturn += "<a href='"+id->not_query+"?page="+j+"'>"+(j+1)+"</a> ";
  }
  toreturn +="</td></tr>";
  toreturn +="</table>";
  return toreturn;
}

What happens if the num_rows function is not available? The same results can be obtained via a simple SQL query, obtained modifying the actual query being executed. It is of course less efficient because two queries are issued instead of one. But it's better than nothing.

The query is obtained replacing the list of fields being fetched with the 'COUNT(*)' SQL function. It has slightly different semantics for complex queries, but for all the query types covered in this manual, it works. You might want to alias it for easier manageability (see ../data_extract/syntax).

So the previous example would have been written as:

#define DBHOST "mysql://user:password@localhost/sample"
#define COUNT_QUERY "select count(*) as num from ids"
#define QUERY "select name, code from ids order by name"
#define ENTRIES_PER_PAGE 10
#define SEEK_IS_BROKEN

string parse (object id)
{
  string toreturn;
  object(Sql.sql) db;
  int number_of_entries, number_of_pages, page, j;
  object(Sql.sql_result) result;
  array(string) row;
  
  page=(int)(id->variables->page);
  toreturn="<table border=1>\n";
  db=Sql.sql(DBHOST);                                 //connect
  number_of_entries=(int)(db->query(COUNT_QUERY)[0]->num); //(1)
  result=db->big_query(QUERY);                        //query
#ifdef SEEK_IS_BROKEN
  //it looks like mysql's implementation of seek() is broken, probably at
  //the mysql level in my version (3.22.29). I'll do a loop to emulate seek
  for (j=0;j<ENTRIES_PER_PAGE*page;j++)
    result->fetch_row();
#else
  result->seek(ENTRIES_PER_PAGE*page);                //skip unneeded results
#endif

  for(j=0; j<10; j++)                  //at most 10 results
  {
    row=result->fetch_row();           //fetch the row              
    if (!row)                          //no more data?
      break;                             //exit
    toreturn += "<tr><td>"+row[0]+"</td><td>"+row[1]+"</td></tr>\n";
  }
                                       //now the links section
  number_of_pages=number_of_entries/ENTRIES_PER_PAGE;
  if (number_of_entries%ENTRIES_PER_PAGE)
    number_of_pages++;                 //there might be an incomplete page
  toreturn+="<tr><td colspan=2>";
  for (j=0;j<number_of_pages;j++)
    toreturn += "<a href='"+id->not_query+"?page="+j+"'>"+(j+1)+"</a> ";
  toreturn +="</td></tr>";
  toreturn +="</table>";
  return toreturn;
}

(1): this line is a quick shortcut using the simpler query (see query) interface. It is appropriate in this case, because the results are tiny. We didn't make any checks on the results either, because their structure is very well-known.

The values returned by fetch_fields depend on the server you are connecting to, save for a few ones which should be always there. This is one of the reasons why you shouldn't need to use this function except during development. Let's see an example of it in action:

With Pike:

> object db=Sql.sql("mysql://user:password@localhost/sample"); Result: object > object res=db->big_query("select country, map_refs, flag from countries"); Result: object > res->fetch_fields(); Result: ({ /* 3 elements */ ([ /* 7 elements */ "decimals":0, "flags":(< /* 2 elements */ "primary_key", "not_null" >), "max_length":2, "length":2, "type":"string", "table":"countries", "name":"country" ]), ([ /* 7 elements */ "decimals":0, "flags":(< /* 1 elements */ "not_null" >), "max_length":1, "length":4, "type":"char", "table":"countries", "name":"map_refs" ]), ([ /* 7 elements */ "decimals":0, "flags":(< /* 2 elements */ "not_null", "blob" >), "max_length":13127, "length":65535, "type":"blob", "table":"countries", "name":"flag" ]) })

An array of mappings is returned, one mapping for each field. The "name" key is always present, as is the "flags" key. The other fields change depending on the server, and (as you might see) on the data type.