Création des Logiciels de gestion d'Entreprise, Création et référencement des sites web, Réseaux et Maintenance, Conception
Création des Logiciels de gestion d'Entreprise, Création et référencement des sites web, Réseaux et Maintenance, Conception
Google BigQuery is designed to make it easy to analyze large amounts of data quickly. Today we announced several updates that give BigQuery the ability to handle arbitrarily large result sets, use window functions for advanced analytics, and cache query results. You are also getting new UI features, larger interactive quotas, and a new convenient tiered pricing scheme. In this post we'll dig further into the technical details of these new features.
BigQuery is able to process terabytes of data, but until today BigQuery could only output up to 128 MB of compressed data per query. Many of you asked for more and from now on BigQuery will be able to output results as large as the largest tables our customers have ever had.
To get this benefit, you should enable the new "--allow_large_results
" flag when issuing a query job, and specify a destination table. All results will be saved to the new specified table (or appended, if the table exists). In the updated web UI these options can be found under the new "Enable Options" menu.
With this feature, you can run big transformations on your tables, plus get big subsets of data to further analyze from the new table.
BigQuery's power is in the ability to interactively run aggregate queries over terabytes of data, but sometimes counts and averages are not enough. That's why BigQuery also lets you calculate quantiles, variance and standard deviation, as well as other advanced functions.
To make BigQuery even more powerful, today we are adding support for window functions (also known as "analytical functions") for ranking, percentiles, and relative row navigation. These new functions give you different ways to rank results, explore distributions and percentiles, and traverse results without the need for a self join.
To introduce these functions with an advanced example, let's use the dataset we collected from the Data Sensing Lab at Google I/O. With the percentile_cont()
function it's easy to get the median temperature over each room:
SELECT percentile_cont(0.5) OVER (PARTITION BY room ORDER BY data) AS median, room
FROM [io_sensor_data.moscone_io13]
WHERE sensortype='temperature'
In this example, each original data row shows the median temperature for each room. To visualize it better, it's a good idea to group all results by room with an outer query:
SELECT MAX(median) AS median, room FROM (
SELECT percentile_cont(0.5) OVER (PARTITION BY room ORDER BY data) AS median, room
FROM [io_sensor_data.moscone_io13]
WHERE sensortype='temperature'
)
GROUP BY room
We can add an additional outer query, to rank the rooms according to which one had the coldest median temperature. We'll use one of the new ranking window functions, dense_rank()
:
SELECT DENSE_RANK() OVER (ORDER BY median) rank, median, room FROM (
SELECT MAX(median) AS median, room FROM (
SELECT percentile_cont(0.5) OVER (PARTITION BY room ORDER BY data) AS median, room
FROM [io_sensor_data.moscone_io13]
WHERE sensortype='temperature'
)
GROUP BY room
)
We've updated the documentation with descriptions and examples for each of the new window functions. Note that they require the OVER()
clause, with an optional PARTITION BY
and sometimes required ORDER BY
arguments. ORDER BY
tells the window function what criteria to use to rank items, while PARTITION BY
allows you to define multiple groups to be analyzed independently of each other.
The window functions don't work with the big GROUP EACH BY
and JOIN EACH BY
operators, but they do work with the traditional GROUP BY
and JOIN BY
. As a reminder, we announced GROUP EACH BY
and JOIN EACH BY
last March, to allow large join and group operations.
BigQuery now remembers values that you've previously computed, saving you time and the cost of recalculating the query. To maintain privacy, queries are cached on a per-user basis. Cached results are only returned for tables that haven't changed since the last query, or for queries that are not dependent on non-deterministic parameters (such as the current time). Reading cached results is free, but each query still counts against the max number of queries per day quota. Query results are kept cached for 24 hours, on a best effort basis. You can disable query caching with the new flag --use_cache
in bq, or "useQueryCache
" in the API. This feature is also accessible with the new query options on the BigQuery Web UI.
The BigQuery UI gets even better: You'll get instant information while writing a query if its syntax is valid. If the syntax is not valid, you'll know where the error is. If the syntax is valid, the UI will inform you how much the query would cost to run. This feature is also available with the bq tool and API, using the --dry_run
flag.
An additional improvement: When running queries on the UI, previously you had to wait until its completion before starting another one. Now you have the option to abandon it, to start working on the next iteration of the query without waiting for the abandoned one.
Starting in July, BigQuery pricing becomes more affordable for everyone: Data storage costs are going from $0.12/GB/month to $0.08/GB/month. And if you are a high-volume user, you'll soon be able to opt-in for tiered query pricing, for even better value.
To support larger workloads we're doubling interactive query quotas for all users, from 200GB + 1 concurrent query, to 400 GB of concurrent queries + 2 additional queries of unlimited size.
These updates make BigQuery a faster, smarter, and even more affordable solution for ad hoc analysis of extremely large datasets. We expect they'll help to scale your projects, and we hope you'll share your use cases with us on Google+.
The BigQuery UI features a collection of public datasets for you to use when trying out these new features. To get started, visit our sign-up page and Quick Start guide. You should take a look at our API docs, and ask questions about BigQuery development on Stack Overflow. Finally, don't forget to give us feedback and join the discussion on our Cloud Platform Developers Google+ page.
<?phpMethods such as query() work just as you’d expect with any MySQL database. This example uses the popular PDO library, although other libraries such as mysql and mysqli work just as well.
$db = new PDO(
'mysql:unix_socket=/cloudsql/hello-php-gae:my-cloudsql-instance;dbname=demo_db;charset=utf8',
'demo_user',
'demo_password'
);
foreach($db->query('SELECT * FROM users') as $row) {
echo $row['username'].' '.$row['first_name']; //etc...
}
<?phpThe same fopen() and fwrite() commands are used just as if you were writing to a local file. The difference is we’ve specified a Google Cloud Storage URL instead of a local filepath.
$handle = fopen('gs://hello-php-gae-files/prime_numbers.txt','w');
fwrite($handle, "2");
for($i = 3; $i <= 2000; $i = $i + 2) {
$j = 2;
while($i % $j != 0) {
if($j > sqrt($i)) {
fwrite($handle, ", ".$i);
break;
}
$j++;
}
}
fclose($handle);
<?php
$primes = explode(",",
file_get_contents('gs://hello-php-gae-files/prime_numbers.txt')
);
if(isset($primes[100]))
echo "The 100th prime number is ".$primes[100];
import googledatastore as datastoreNext include writeEntity() and readEntity() functions:
def main()
writeEntity()
readEntity()
def WriteEntity():First create a new file called “demo.py”. Inside demo.py, we’ll add code to write and then read an entity from the Cloud Datastore. Finally we can update main() to print out the property values within the fetched entity:
req = datastore.BlindWriteRequest()
entity = req.mutation.upsert.add()
path = entity.key.path_element.add()
path.kind = 'Greeting'
path.name = 'foo'
message = entity.property.add()
message.name = 'message'
value = message.value.add()
value.string_value = 'to the cloud and beyond!'
try:
datastore.blind_write(req)
except datastore.RPCError as e:
# remember to do something useful with the exception pass
def ReadEntity():
req = datastore.LookupRequest()
key = req.key.add()
path = key.path_element.add()
path.kind = 'Greeting0'
path.name = 'foo0'
try:
resp = datastore.lookup(req)
return resp
except datastore.RPCError as e:
# remember to do something useful with the exception pass
def main()Before we can run this code we need to tell the SDK which Cloud Datastore instance we would like to use. This is done by exporting the following environment variable:
writeEntity();
resp = readEntity();
entity = resp.found[0].entity
for p in entity.property:
print 'Entity property name: %s', p.name
v = p.value[0]
print 'Entity property value: %s', v.string_value
~$ export DATASTORE_DATASET cloud-datastore-demoFinally we’re able to run the application by simply issuing the following:
~$ python demo.pyBesides the output that we see in console window, we’re also able to monitor our interactions within the Cloud Console. By navigating back to Cloud Console, selecting our cloud-datastore-demo project, and then selecting the Cloud Datastore we’re taken to our instance’s dashboard page that includes number of entities, properties, and property types, as well as index management, ad-hoc query support and breakdown of stored data.
Starting today, Google Compute Engine is available to all customers who sign up for our Gold Support package. We’re also happy to announce a 4% reduction on all Compute Engine pricing.
In the nine months since announcing Compute Engine, customers have been using Google’s Infrastructure as a Service product and giving us valuable feedback. Sebastian Stadil of Scalr wrote, in a recent review:
“Google Compute Engine is not just fast. It’s Google fast. In fact, it’s a class of fast that enables new service architectures entirely.”
We’re happy to hear that, because one of our main goals in building Compute Engine is to enable a new generation of applications with direct access to the capabilities of Google’s vast computing infrastructure.
Based on user feedback, we’ve added a number of major features including:
While we've been hard at work developing new features, we've also had the opportunity to play. Check out the amazing World Wide Maze Chrome Experiment, developed by the Chrome team in Japan. This game converts any web site of your choice into an interactive, three dimensional maze, navigated remotely via your smartphone. Compute Engine virtual machines run Node.js to manage the game state and synchronization with the mobile device, while Google App Engine hosts the game’s web UI. This application provides an excellent example of the new kinds of rich, high performance back end services enabled by Google Cloud Platform.
With today’s announcement, we look forward to welcoming many new customers, and bringing exciting new applications to Google Cloud Platform!
Have you ever wanted to integrate SMS or voice communications into your app? We’ve been working with our friends over at Twilio to make it easier to do so. Today we’re announcing native Python and Java libraries for working with Twilio APIs onto Google Cloud Platform.
Lots of apps on App Engine have already been built with phone functionality. Check out the sample code for a group messaging app and the sample code for an app that dispatches voicemails and SMS messages to PagerDuty. Learn how to send business cards via sms through this step by step guide.
You can start building voice and SMS features into your App Engine apps today. Together with Twilio, we’ll help you get started with 2,000 free text message or voice minutes.
Ready to get started?
Here’s a quick peek at how easy it can be to send a text message from App Engine using Python. After installing the Twilio library, it just takes a few lines of code to send an SMS.
import webapp2
from twilio import twiml
from twilio.rest import TwilioRestClient
class SendSMS(webapp2.RequestHandler):
def get(self):
# replace with your credentials from: https://www.twilio.com/user/account
account_sid = "ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
auth_token = "xxxxxxxxxxxxxxxxxxxxxxxxxx"
client = TwilioRestClient(account_sid, auth_token)
# replace "to" and "from_" with real numbers
rv = client.sms.messages.create(to="+14155551212",
from_="+14085551212",
body="Hello Monkey!")
self.response.write(str(rv))
app = webapp2.WSGIApplication([('/send_sms', SendSMS)],
debug=True)
/* JOIN EACH example * Selects the top 10 most edited Wikipedia pages * of words that appear in works of Shakespeare. */ SELECT TOP(wiki.title, 10), COUNT(*) FROM [publicdata:samples.wikipedia] AS wiki JOIN EACH [publicdata:samples.shakespeare] AS shakespeare ON shakespeare.word = wiki.title; |
/* TIMESTAMP example * Which hours in the day are the most popular for GitHub actions? * This query converts github_timeline "created_at" date time * strings to BigQuery TIMESTAMP, and extracts the hour from each. */ SELECT HOUR(TIMESTAMP(created_at)) AS event_create_hour, COUNT(*) AS event_count FROM [publicdata:samples.github_timeline] GROUP BY event_create_hour ORDER BY event_count DESC; |
(Cross-posted with the App Engine and Enterprise Blogs)
Support is as important as product features when choosing a platform for your applications. And let’s face it, sometimes we all need a bit of help. No matter which Google Cloud Platform services you are using — App Engine, Compute Engine, Cloud Storage, Cloud SQL, BigQuery, etc. — or what time of day, you should be able to get the answers you need. While you can go to Stack Overflow or Google Groups, we realize some of you may need 24x7 coverage, phone support or direct access to a Technical Account Manager team.
To meet your support requirements, we’re introducing a comprehensive collection of support packages for services on Google Cloud Platform, so you can decide what level best fits your needs:
Sign up or click here to find out more information about the new Google Cloud Platform support options.
Brett McCully is the Manager of the Google Cloud Platform Support team and is currently based in Seattle.
Posted by Ashleigh Rentz, Editor Emerita
2013, By: Seo MasterTry-with-resources, which helps avoid memory leaks and related bugs by automatically closing resources that are used in a try-catch statement.
public static void invokeExample() {
String s;
MethodType mt;
MethodHandle mh;
MethodHandles.Lookup lookup = MethodHandles.lookup();
MethodType mt = MethodType.methodType(String.class, char.class,
char.class);
MethodHandle mh = lookup.findVirtual(String.class, "replace", mt);
s = (String) mh.invokeExact("App Engine Java 6 runtime",'6','7');
System.out.println(s);
}
Flexible Type Creation when using generics, enabling you to create parameterized types more succinctly. For example, you can write:
public static void viewTable(Connection con, String query) throws SQLException {
try (
Statement stmt = con.createStatement();
ResultSet rs = stmt.executeQuery(query)
) {
while (rs.next()) {
// process results
//
}
} catch (SQLException e) {
// con resource is auto-closed, no need to do anything here!
//
}
}
instead of:
Map<String, List<String>> myMap = new HashMap<>();
Map<String, List<String>> myMap = new HashMap<String, List<String>>();In addition to the language features listed above, the App Engine Java 7 runtime also includes:
Then, expose it over a standard REST interface with a simple attribute and a versioning pattern.
public class SuperHeroes {
public ListlistSuperHeroes() {
Listlist = new ArrayList ();
list.add(new SuperHero ("Champion of the Obvious", "Brad Abrams"));
list.add(new SuperHero ("Mr. Justice", "Chris Ramsdale"));
return list;
}
}
Now you have a simple REST interface.
@Api(name = "superheroes", version = "v1")
public class SuperHeroesV1 {
...
}
And you can make strongly typed calls from your Android clients:
$ curl http://localhost:8888/_ah/api/superheroes/v1/superheroes
{
"items": [
{
"knownAs" : "Champion of the Obvious",
"realName" : "Brad Abrams"
},
{
"knownAs" : "Mr. Justice",
"realName" : "Chris Ramsdale"
}
Or Objective-C iOS client:
Real result = superheroes.list().execute();
Or the web client in JavaScript:
GTLQuerySuperHeroesV1 *query = [GTLQuerySuperHeroesV1 queryForSuperHeroesList];
[service executeQuery:query completionHandler:^(GTLServiceTicket *ticket,
GTLSuperHeroes *object, NSError *error) {
NSArray *items = [object items];
}];
// ...Read the documentation for Java or Python to discover how you can build a simple tic-tac-toe game using Cloud Endpoints.
var ROOT = 'https://' + window.location.host + '/_ah/api';
gapi.client.load('superheroes', 'v1',
loadSuperheroesCallback, ROOT);
// Get the list of superheroes
gapi.client.superheroes.superheroes.list().execute(function(resp) {
displaySuperheroesList(resp);
});