Hubdog Preview: Google Watch

| register

adult filtering : on

new channels

suggestions

| most popular

| help

| fetch

From the same publisher

Google Watch

Not yet rated

Google, the World, and the World Wide Web, Weblogged

Hubdog allows you to consume this channel directly from your Windows Mobile

Google Watch

Google, the World, and the World Wide Web, Weblogged

Sorting 1PB with MapReduce

At Google we are fanatical about organizing the world's information. As a result, we spend a lot of time finding better ways to sort information using MapReduce, a key component of our software infrastructure that allows us to run multiple processes simultaneously. MapReduce is a perfect solution for many of the computations we run daily, due in large part to its simplicity, applicability to a wide range of real-world computing tasks, and natural translation to highly scalable distributed implementations that harness the power of thousands of computers.

In our sorting experiments we have followed the rules of a standard terabyte (TB) sort benchmark. Standardized experiments help us understand and compare the benefits of various technologies and also add a competitive spirit. You can think of it as an Olympic event for computations. By pushing the boundaries of these types of programs, we learn about the limitations of current technologies as well as the lessons useful in designing next generation computing platforms. This, in turn, should help everyone have faster access to higher-quality information.

We are excited to announce we were able to sort 1TB (stored on the Google File System as 10 billion 100-byte records in uncompressed text files) on 1,000 computers in 68 seconds. By comparison, the previous 1TB sorting record is 209 seconds on 910 computers.

Sometimes you need to sort more than a terabyte, so we were curious to find out what happens when you sort more and gave one petabyte (PB) a try. One petabyte is a thousand terabytes, or, to put this amount in perspective, it is 12 times the amount of archived web data in the U.S. Library of Congress as of May 2008. In comparison, consider that the aggregate size of data processed by all instances of MapReduce at Google was on average 20PB per day in January 2008.

It took six hours and two minutes to sort 1PB (10 trillion 100-byte records) on 4,000 computers. We're not aware of any other sorting experiment at this scale and are obviously very excited to be able to process so much data so quickly.

An interesting question came up while running experiments at such a scale: Where do you put 1PB of sorted data? We were writing it to 48,000 hard drives (we did not use the full capacity of these disks, though), and every time we ran our sort, at least one of our disks managed to break (this is not surprising at all given the duration of the test, the number of disks involved, and the expected lifetime of hard disks). To make sure we kept our sorted petabyte safe, we asked the Google File System to write three copies of each file to three different disks.

Significantly improved handling of the so-called "stragglers" (parts of computation that run slower than expected) was a key software technique that helped sort 1PB. And of course, there are many other factors that contributed to the result. We'll be discussing all of this and more in an upcoming publication. And you can also check out the video from our recent Technology RoundTable Series.

Posted by Grzegorz Czajkowski, Systems Infrastructure Team

Color Soup, a Collaborative Mosaic

“Colorful soup” – BunteSuppe.de – is the name of a German collaborative art tool. Just drag & drop any tile on the screen to create a larger picture.

[By Philipp Lenssen | Origin: Color Soup, a Collaborative Mosaic | Comments]

[Advertisement] Want to advertise here? Your ad will show in the blog and feed.

Our international approach to search

In previous posts in this series, you have read about the challenges of building a world-class search engine. Our goal is to make Google’s search be relevant to all people, regardless of their language or country. As my colleague Amit Singhal described, we use statistical data as the basis for making sweeping algorithmic changes. Many of these changes can be rolled out across all languages we support, but in some cases the unique characteristics of each language require some algorithmic considerations and tuning. And to make things really interesting, there are cases where the same language is different across countries. Obvious examples are "color" in the U.S. vs. "colour" in the U.K., or "camião" in Portugal vs. "caminhão" in Brazil.

My name is Daphne Dembo, and my focus is improving Google's international search. This is a tough challenge, since Google search is used in many countries and languages where our engineers have little personal knowledge. Initially, the international search improvements were done by Search Quality engineers who were passionate about their languages and countries: Lina from Sweden improved our parsing of compound words in German and Swedish; Dimitra from Greece introduced diacritical support; Ishai from Israel worked on transliteration corrections for Hebrew and Arabic; Trystan from Australia created methods for identifying local search results and ranking them together with foreign ones from the same language; Alex, a bilingual Ukrainian and Russian, introduced morphological understanding of these languages. As the importance of our international search grew, we solicited help from Googlers in all our offices. Finally, we are leveraging an international network of search specialists who help us understand search within the unique combination of their language and country.

Our first step in providing search support for a language is to train our language model on a large collection of documents in that language. This ensures that our language model is more precise and comprehensive — for example, it incorporates names, idioms, colloquial usage, and newly coined words not often found in static dictionaries. For instance, we recently started identifying Swahili, and used pages such as this one for the Parliament of Tanzania to train our system with the language's nuances. Having a trained language model helps to categorize documents during crawling and indexing of the web and to parse the user's query. Once this stage was complete, we launched Swahili search in countries such as Tanzania and Kenya, enabling local searches for the "Dar es Salaam stock exchange" [Soko la hisa dar es salaam], and "cure for Malaria" [Tiba ya malaria]. (As always, we are using square brackets to denote a search query. For example, you can search for "soccer" in Hamburg, Germany by clicking on [fußball in hamburg]).

We learn some things from our users, so as people start using our search engine, we can improve the way we rank in that language. Here are few examples:

Spell corrections: We recently launched spell corrections in Estonian. If your Estonian is rusty, and you don't remember how to spell "smoke detector," we can suggest a spell correction for [suitsuantur], leading to better search results.
Diacritical marks: Many languages have diacritical marks, which alter pronunciation. Our algorithms are built to support them, and even help users who mis-type or completely ignore them. For example, if you're a resident of Quebec, Canada and would like to know the weather forecast in Quebec City, we'll serve good results whether you type with diacritical signs [Météo à Québec] or without [meteo quebec]. Czech users can read the same excellent results for a popular kids' cartoon by searching for [krtecek] and [krteček]. On the other hand, sometimes diacriticals change the meaning of the word and we have to use them correctly. For example, in Thai, [ข้าว] is "rice," with completely different results than [ข่าว], which is "news"; or in Slovakia, results for "child" [dieťa] are different than results for "diet" [diéta].
Synonyms: A general case of diacritical support is the handling of synonyms in different languages. Korean searches showed that "samsung" can be viewed as a synonym of "삼성", so that when users search for [samsung], they find results which have the company's name in Korean.
Compounding: Some languages allow compounding, which is the formation of new words by combining together existing words. You can see a nice example in Swedish, where we return documents about a Swedish credit card for both compounded [Visakort] and non-compounded [visa kort] queries.
Stemming: Google has developed morphological models that can receive compound words as queries, and return pages which contain their stem, possibly as part of a different compound. For example, when searching for cars in Saudi Arabia, you can search for [سيارة] and [سيارات] because both are variants of the same stem, and both return many common results. A Polish user can search for "movie" [film], and get back results that contain other variants of the stem, such as "filmów," "filmu," "filmie," "filmy." A user from Belarus will find results for all word forms of the capital, Minsk [Мінск]: "Мінску," "Мінска," "Мінскага."

In addition to these semantic factors, Google does even more to parse documents and queries. Understanding the details of language usage in a country is important. Notation of acronyms is different across languages: In Hebrew it is double quotes before the last (left-most) character, as in "prime minister" [רה"מ]; in Thai — a dot at the end of the word, as in police station [สน. ]; while in the U.S. — dots after each character, as in [I.B.M.]. Chinese users quote works of art with a "《", as in: [《手机》剧情], and denote dates with a "日", as in: [2006年1月13日].

Beyond the linguistic elements of a language, we consider how people enter a query. For example, some languages that do not have Latin scripts require keyboards with dual alphanumeric keys. The user can switch between language input modes by typing special keystrokes. In case the user forgets to type this sequence, the queries end up being gibberish. You can see correct handling of these mistakes in Arabic ([hgsuv] corrected to [السعر]) and ([حقثسهيثىفهشم ثممثؤفهخىس ] corrected to [presidential elections]), Hebrew ([vdrk, kuyu] corrected to [הגרלת לוטו]), and Cyrillic ([rehc ljkffhf] corrected to [курс доллара]).

Another way of avoiding the inconvenience of switching keyboard modes is by typing the phonetic sounds of the query in Latin characters. Recreating the correct query in the target language isn't trivial, since there might be many possibilities. We can see several such examples in which we suggest the same query in the intended language for Russian ([biskvitnyi rulet] to [бисквитный рулет]), "movies" in Chinese ([dianying] to [电影]), and "Bank of Attica" in Greek [trapeza attikhs] returns good results for "Τράπεζα Αττικής". Users of 8 Indic languages (such as Hindi, Gujarati, Telugu) can type the phonetic sound of the query, and choose the words in Hindi script:

Ease of typing and reading is also influenced by the language used. Since every Chinese word requires several keystrokes on a standard keyboard, we provide category browsing by Images and related searches so that people don't need to type as much. Similarly, we are now launching Google Suggest, or real-time completion of queries, in many languages.

So far I described how we improve the quality of search in a language. However, there is a strong effect of the location of the user, even if it is only approximated to the country, since in many cases local content is more relevant than global information. For example, searching for Spanish Yellow Pages [Páginas Amarillas] will result in several documents of global interest and several local results in Peru, Mexico, and Spain. Similar to that, searching for [Côte d'Or] in France will return results for that region, whereas searches in Belgium will return results about the chocolate maker.

Note that the display of information should conform to the standards in that country, so we display "," as a decimal notation for Croatian users who want to know how many millimeters are in an inch [inč u milimetrima], or for Italian users who are interested in currency exchange rates [50 euro in dollari]. Similarly, temperatures in Norway [Været i Oslo] will be displayed in Celsius, while in the U.S. — in Fahrenheit [weather Boston].

If everything else fails, we provide cross-language translations based upon Google's translation technology described in this blog post. We will translate your query to English, search English documents on the web, and translate the returned results from English back into the original query language. For example, Japanese users who are interested in viewing Halloween illustrations (Halloween is a holiday which originated in Ireland) can search for [ハロウィンイラスト]. You can then request a Japanese translation of the English pages (at the bottom of the page), which will bring up the translation page in the screenshot below. Similarly, Korean users can search for the latest on Harry Potter [해리 포터], and Arabic readers can search for the opening of the Sydney Opera house [افتتاح دار الاوبرا في سيدني]. (Click on the image to see a larger version.)

All in all, Google Search is being actively developed for more than 100 languages, in 150+ countries, with dozens of improvements launched each month. So far I've covered the basics of how international search works, but this is just the surface of all the international work we do. There are many other interesting topics that impact international markets like usability, homepage and results page layout, and connectivity. An understanding of real cultural and human factors is essential to creating a search engine that resonates with the people who use it. (Click on the image to see a larger version.)

(Update: Replaced example in the 4th bullet point.)

Posted by Daphne Dembo, Engineering Director

Google Now Lets You Upvote Results and Comment On Them

Google has gone live with a big change to their result pages, at least for those of you who are logged in (if you’re not seeing it yet, it may still be rolled out for your Google Account). It’s called SearchWiki, and lets you edit the position of the results you’re getting, and add comments to them. SearchWiki was in experimental stage for some time now.

Specifically, you’ll be seeing three icons accompanying results, and further options below the listing:

Up vote: An up arrow, similar in functionality to what you may know from social sites like Reddit or Digg. Clicking it will turn the icon green and move this specific result up one position. Once upped, a down arrow appears as well, which will trigger the result to fly to the bottom of the listing. (At his blog, Ionut Alex. Chitu mentions: “[Y]our changes are available only when you repeat the query and, in some cases, for similar queries (e.g.: [google.com] in addition to [google]). That means you can’t remove a web page or a domain from all search results”.)
Remove: An X icon, which will make the result disappear in an animated puff. It won’t be completely gone for you, though; at the bottom of the page you’ll see the note “You have removed results from this page” with an option to hide them altogether, or restore them.
Comment: A Speech Bubble icon which lets you make a comment on the result. The comment will be public, Google disclaims. Once saved, you’ll still be able to edit or delete your comment later on. Others are now able to upvote your comment or flag it as innapropriate, like on the “All SearchWiki notes” page. (That page also serves as the next best thing to see the pure vote-based ranking.)
Add result: The plus icon is shown below the organic results, and it lets you add any URL at all to your result page.

Now, when you change something, you won’t immediately shift around the page for others. For now Google says it’s a mere customization on your end. (You can see all your customizations in one place at the “My SearchWiki notes” page.) However, Google indicates in statements provided to Search Engine Land that they won’t completely rule out the possibility of this impacting everyone’s rankings in the future:

<<I asked what would happen if 10,000 people all added “Matt McGee’s Widget Page” to their own results for the phrase [widget]. “We’re always looking at user data as a signal,” [Google’s Cedric Dupont] says. And in a situation like that? “We’re not closing any doors.">>

Also, once a result was upvoted, you’ll be seeing who else voted for this result, though it will only show compactly as e.g. “ [up] 9 11 - Picked by Rat, Mr, yinan.wu, and others.” This may add a more social feeling to search results. (Google calls it a “community” in their announcement post on this, but we need to keep in mind how diverse this group is, even when they might have stumbled upon the same pages in results.) Note this field won’t show your full email address to others, but your nickname, which you can change on your account profile page.

It’s probably also not a huge jump to imagine that Google could one day extract keywords from the comments of a particular result to aid them in their results selection for exotic queries. And as opposed to a web index, which at least in theory anyone with enough servers could build, the upvotes, hides and comment data is something Google will exclusively own thanks to their (massive) user base.

Now, all these new features come with a certain amount of clutter, naturally. Ionut in the comments remarks, “Google should provide a separate wiki mode (placing a link like ’edit search results’, ’change the results’) that adds voting buttons, commenting options.” I guess doing so wouldn’t get as many people to participate though – which for Google could decrease the valuable crowd intelligence they may tap with this move.

[Thanks Russell O., Tony and Oradzuza!]

[By Philipp Lenssen | Origin: Google Now Lets You Upvote Results and Commen ... | Comments]

[Advertisement] Find the right keywords for your campaigns at KeywordDiscovery.com

SearchWiki: make search your own

Have you ever wanted to mark up Google search results? Maybe you're an avid hiker and the trail map site you always go to is in the 4th or 5th position and you want to move it to the top. Or perhaps it's not there at all and you'd like to add it. Or maybe you'd like to add some notes about what you found on that site and why you thought it was useful. Starting today you can do all this and tailor Google search results to best meet your needs.

Today we're launching SearchWiki, a way for you to customize search by re-ranking, deleting, adding, and commenting on search results. With just a single click you can move the results you like to the top or add a new site. You can also write notes attached to a particular site and remove results that you don't feel belong. These modifications will be shown to you every time you do the same search in the future. SearchWiki is available to signed-in Google users. We store your changes in your Google Account. If you are wondering if you are signed in, you can always check by noting if your username appears in the upper right-hand side of the page.

The changes you make only affect your own searches. But SearchWiki also is a great way to share your insights with other searchers. You can see how the community has collectively edited the search results by clicking on the "See all notes for this SearchWiki" link.

Watch our lead engineer, Amay, demonstrate a few ways to use SearchWiki in this short video:

This new feature is an example of how search is becoming increasingly dynamic, giving people tools that make search even more useful to them in their daily lives. We have been testing bits and pieces of SearchWiki for some time through live experiments, and we incorporated much of our learnings into this release. We are constantly striving to improve our users' search experience, and this is yet another step along the way.

Posted by Cedric Dupont, Product Manager, and Corin Anderson, Software Engineer

Not yet rated

Reviews: