Direktori 103: December 2009

Leap day hackathon for Google Gadgets, Maps, and more

Written by Ben Lisbakken, Google Developer Programs

If you've got JavaScript skills and you'd like to implement such things as Google Gadgets or Maps on your site, bring your laptops and come hang out with us in Mountain View.

This Friday, my team (Google Developer Programs) is hosting a hackathon to get you started with our JavaScript APIs. There will be plenty of our engineers around to answer questions. We'll start with short introductions of the APIs and then break into groups for coding and camaraderie. There'll be food, and prizes too.

The featured JavaScript APIs:Google Gears -- make your web applications work offlineGoogle AJAX Search/Feeds -- add Google search capabilities to your websiteGoogle Gadgets -- create small programs that run on your iGoogle homepage
Google Maps -- embed and manipulate Google Maps on your web pageGoogle Calendar -- interact with Google Calendars on your websiteWhen: Friday, February 29 - two sessions (you're welcome to attend both)2-5:30 PM6-10 PM Where: The Googleplex
Building 40
1600 Amphitheatre Pkwy
Mountain View, CA 94043
Room: Seville Tech Talk, 2nd floor

See our map for parking locations and where to check in. (Soon, you too, will be making maps like this! :)

Just say yes and RSVP!

And no worries if you're busy this Friday; future hackathons will feature other APIs and more languages. Check out the Developer Events Calendar for future listings. Hope to see you soon.

Speaking the language of robots

Written by Sean Harding, Webmaster Tools Team

We all know how friendly Googlebot is. And like all benevolent robots, he listens to us and respects our wishes about parts of our site that we don't want crawled. We can just give him a robots.txt file explaining what we want, and he'll happily comply. But what if you're intimidated by the idea of communicating directly with Googlebot? After all, not all of us are fluent in the language of robots.txt. This is why we're pleased to introduce you to your personal robot translator: the Robots.txt Generator in Webmaster Tools. It's designed to give you an easy and interactive way to build a robots.txt file. It can be as simple as entering the files and directories you don't want crawled by any robots.

Or, if you need to, you can create fine-grained rules for specific robots and areas of your site.

Once you're finished with the generator, feel free to test the effects of your new robots.txt file with our robots.txt analysis tool. When you're done, just save the generated file to the top level (root) directory of your site, and you're good to go. There are a couple of important things to keep in mind about robots.txt files: Not every search engine will support every extension to robots.txt files The Robots.txt Generator creates files that Googlebot will understand, and most other major robots will understand them too. But it's possible that some robots won't understand all of the robots.txt features that the generator uses. Robots.txt is simply a request Although it's highly unlikely from a major search engine, there are some unscrupulous robots that may ignore the contents of robots.txt and crawl blocked areas anyway. If you have sensitive content that you need to protect completely, you should put it behind password protection rather than relying on robots.txt.

We hope this new tool helps you communicate your wishes to Googlebot and other robots that visit your site. If you want to learn more about robots.txt files, check out our Help Center. And if you'd like to discuss robots.txt and robots with other webmasters, visit our Google Webmaster Help Group.

First date with the Googlebot: Headers and compression

Written by Maile Ohye as the website, Jeremy Lilley as the Googlebot

Name/User-Agent: Googlebot
IP Address: Verify it here
Looking For: Websites with unique and compelling content
Major Turn Off: Violations of the Webmaster Guidelines
Googlebot -- what a dreamboat. It's like he knows us , , and soul. He's probably not looking for anything exclusive; he sees billions of other sites (though we share our data with other bots as well :), but tonight we'll really get to know each other as website and crawler.

I know, it's never good to over-analyze a first date. I'm going to get to know Googlebot a bit more slowly, in a series of posts:
Our first date (tonight!): Headers Googlebot sends, file formats he "notices," whether it's better to compress dataJudging his response: Response codes (301s, 302s), how he handles redirects and If-Modified-SinceNext steps: Following links, having him crawl faster or slower (so he doesn't come on too strong)And tonight is just the first date...

***************
Googlebot: ACK
Website: Googlebot, you're here!
Googlebot: I am.

GET / HTTP/1.1
Host: example.com
Connection: Keep-alive
Accept: */*
From: googlebot(at)googlebot.com
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Accept-Encoding: gzip,deflate

Website: Those headers are so flashy! Would you crawl with the same headers if my site were in the U.S., Asia or Europe? Do you ever use different headers?

Googlebot: My headers are typically consistent world-wide. I'm trying to see what a page looks like for the default language and settings for the site. Sometimes the User-Agent is different, for instance AdSense fetches use "Mediapartners-Google":
  User-Agent: Mediapartners-Google

Or for image search:
  User-Agent: Googlebot-Image/1.0

Wireless fetches often have carrier-specific user agents, whereas Google Reader RSS fetches include extra info such as number of subscribers.

I usually avoid cookies (so no "Cookie:" header) since I don't want the content affected too much by session-specific info. And, if a server uses a session id in a dynamic URL rather than a cookie, I can usually figure this out, so that I don't end up crawling your same page a million times with a million different session ids.

Website: I'm very complex. I have many file types. Your headers say "Accept: */*". Do you index all URLs or are certain file extensions automatically filtered?

Googlebot: That depends on what I'm looking for.

If I'm indexing for regular web search, and I see links to MP3s and videos, I probably won't download those. Similarly, if I see a JPG, I will treat it differently than an HTML or PDF link. For instance, JPG is much less likely to change frequently than HTML, so I will check the JPG for changes less often to save bandwidth. Meanwhile, if I'm looking for links as Google Scholar, I'm going to be far more interested in the PDF article than the JPG file. Downloading doodles (like JPGs) and videos of skateboarding dogs is distracting for a scholar—do you agree?

Website: Yes, they can be distracting. I'm in awe of your dedication. I love doodles (JPGs) and find them hard to resist.

Googlebot: Me, too; I'm not always so scholarly. When I crawl for image search, I'm very interested in JPGs. And for news, I'm mostly looking at HTML and nearby images.

There are also plenty of extensions (exe, dll, zip, dmg...), that tend to be big and less useful for a search engine.

Website: If you saw my URL, http://www.example.com/page1.LOL111, would you (whimper whimper) reject it just because it contains an unknown file extension?

Googlebot: Website, let me give a bit more background. After actually downloading a file, I use the Content-Type header to check whether it really is HTML, an image, text, or something else. If it's a special data type like a PDF file, Word document, or Excel spreadsheet, I'll make sure it's in the valid format and extract the text content. Maybe it has a virus; you never know. If the document or data type is really garbled, there's usually not much to do besides discard the content.

So, if I'm crawling http://www.example.com/page1.LOL111 with an unknown file extension, it's likely that I would start to download it. If I can't figure out the content type from the header, or it's a format that we don't index (e.g. mp3), then it'll be put aside. Otherwise, we proceed indexing the file.

Website: My apologies for scrutinizing your style, Googlebot, but I noticed your Accept-Encoding headers say:
Accept-Encoding: gzip,deflate

Can you explain these headers to me?

Googlebot: Sure. All major search engines and web browsers support gzip compression for content to save bandwidth. Other entries that you might see here include "x-gzip" (the same as "gzip"), "deflate" (which we also support), and "identity" (none).

Website: Can you talk more about file compression and "Accept-Encoding: gzip,deflate"? Many of my URLs consist of big Flash files and stunning images, not just HTML. Would it help you to crawl faster if I compressed my larger files?

Googlebot: There's not a simple answer to this question. First of all, many file formats, such as swf (Flash), jpg, png, gif, and pdf are already compressed (there are also specialized Flash optimizers).

Website: Perhaps I've been compressing my Flash files and I didn't even know? I'm obviously very efficient.

Googlebot: Both Apache and IIS have options to enable gzip and deflate compression, though there's a CPU cost involved for the bandwidth saved. Typically, it's only enabled for easily compressible text HTML/CSS/PHP content. And it only gets used if the user's browser or I (a search engine crawler) allow it. Personally, I prefer "gzip" over "deflate". Gzip is a slightly more robust encoding — there is consistently a checksum and a full header, giving me less guess-work than with deflate. Otherwise they're very similar compression algorithms.

If you have some spare CPU on your servers, it might be worth experimenting with compression (links: Apache, IIS). But, if you're serving dynamic content and your servers are already heavily CPU loaded, you might want to hold off.

Website: Great information. I'm really glad you came tonight — thank goodness my robots.txt allowed it. That file can be like an over-protective parent!

Googlebot: Ah yes; meeting the parents, the robots.txt. I've met plenty of crazy ones. Some are really just HTML error pages rather than valid robots.txt. Some have infinite redirects all over the place, maybe to totally unrelated sites, while others are just huge and have thousands of different URLs listed individually. Here's one unfortunate pattern. The site is normally eager for me to crawl:
  User-Agent: *
  Allow: /

Then, during a peak time with high user traffic, the site switches the robots.txt to something restrictive:
  # Can you go away for a while? I'll let you back
  # again in the future. Really, I promise!
  User-Agent: *
  Disallow: /

The problem with the above robots.txt file-swapping is that once I see the restrictive robots.txt, I may have to start throwing away content I've already crawled in the index. And then I have to recrawl a lot of content once I'm allowed to hit the site again. At least a 503 response code would've been temporary.

I typically only re-check robots.txt once a day (otherwise on many virtual hosting sites, I'd be spending a large fraction of my fetches just getting robots.txt, and no date wants to "meet the parents" that often). For webmasters, trying to control crawl rate through robots.txt swapping usually backfires. It's better to set the rate to "slower" in Webmaster Tools.

Googlebot: Website, thanks for all of your questions, you've been wonderful, but I'm going to have to say "FIN, my love."

Website: Oh, Googlebot... ACK/FIN. :)

***************

http://business-catalog.blogspot.com/

Help you find the best way to earn money Online!
- SFI-The World's Largest Affiliate Business Network.
- Eight ways to Earn with ACME people search.

Website: http://business-catalog.blogspot.com/

http://www.backlink4u.net/

International Backlinks

International Link Exchange, Free automatic backlink, inbound link, free traffic booster, submit site, ping update, google page rank checker, sitemap submitter, free PHP. Automatic Backlinks for optimizing blog or website on search engine and get more high PageRanks.

Website: http://www.backlink4u.net

http://resep-ok.blogspot.com/

World Wide Recipes

Kumpulan Resep Masakan Indonesia dan Internasional.
ayam goreng lapis, ayam betutu, masak habang, chicken paella, ayam goreng mbok berek, ayam goreng mbok berek II, ayam pop, rendang ayam, botok ayam, ayam goreng terasi, ayam rica-rica, ayam lombok, ayam goreng kuning, ayam opor, pelecing ayam, ayam kecap jeruk, ayam goreng bumbu bali, ayam bumbu bali, ayam goreng kalasan, ayam goreng mentega, ayam goreng kalasan keringan, ayam bakar padang, paru goreng kering serundeng, besengek daging, malbi daging sapi, abon daging sapi, abon sapi, kresengan, sate manis, baso, gepuk daging, stup steak campur, sate merah, steak saus black pepper cheese, sate bumbu kemiri, sambal goreng paru, nasi kebuli, paru goreng kering, kua asang rusuk, omelet daging, rendang, kambing guling, serapah daging (bali), sate bumbu kacang, gulai gazebo, bakso, rib skewer, iga bakar petis, asam-asam ikan gabus, ikan kuah asam, gulai kepala ikan, ikan asin pedas, bandeng presto, ikan laut bakar bumbu kuning, ikan bumbu tauco, ikan goreng tepung, ikan bakar ala raja, pangek ikan, otak-otak pelepah jamur, kakap asem manis, pesmol, gulai kepala ikan kakap merah, woku ikan dibelanga, bandeng bumbu bali, masak ikan dibotoh, sasate bandeng (jawa barat), otak-otak ikan tenggiri, tekwan, siomay, grilled salmon rainbow sauce, nasi kebuli, nasi goreng kunyit, nasi hainan, gudeg, tumis bunga pepaya, tumis campur, sayur jagung manis, perkedel kemangi, sayur gurih buncis, soto mie udang, bakwan surabaya, mie petis siram, mi kuning siam, spaghetti bumbu bali, mie ayam bakso, mie ayam, kepiting asam pedas, udang pancet lado, palai udang segar, tom yam seafood, empek-empek palembang, telur petis lading, martabak telor, martabak telor 2, ketupat tahu, tahu gejrot, tumis tahu daging giling, bumbu dasar merah, bumbu dasar putih, bumbu dasar oranye, bumbu dasar kuning, bolu kenari kayumanis, puding busa coklat, roti goreng, puding karamel, tart kukus, kue nagasari, bingke siram, perut ayam, kue ruwok, kue mako, kue lumpur, sarikaya, kue kering kacang kismis, kue keju keripik jagung, kue mangkok, donat, bolu kukus spekoek, kroket, lapis surabaya, bika ambon, black forest cake, sus layars strawberry, bolu lapis aprikot, cashewnut balls, panca rasa juice, es alpukat, es teler, stamina juice, es jeruk dengan buah, es pelangi, es agar-agar, souffle kopi dingin, biji salak, pineapple milkshake, mango milkshake, easy Shrimp skillet, curried tofu salad, cheeseburger joes, tomato-cucumber raita, crescent samosas, thai-style beef salad over angel-hair pasta, spiced beef pho with sesame chile oil, sukiyaki, real deal chili, texas toasts with filet mignon, watercress and herb butter, beef noodle bowl, inside-out cheeseburgers, rigatoni with beef and eggplant ragu, eatingWell's picadillo, linguine with squash noodles and pine nuts, kootu curry, spaghetti with zesty bolognese, taco salad made over, california rolls, eggplant salad with lemon-flavored plum dressing, marinated bocconcini, snapper ceviche with childs and herbs, green beans in sesame dressing, rainbow pizza, yakiniku, gyoza, yakisoba, sushi cut and handrolls, moroccan-rubbed grilled steak and sweet potatoes, smothered sirloin steak in parmesan-peppercorn sauce, skirt steak with cilantro garlic sauce, beef chow mein, beef curry, sweet and sour meatball simmer, beef satays over thai salad, pecan-crusted beef tenderloin with juniper jus, korean-style steak and lettuce wraps, beef keema, beef and potato stroganoff, beef stew with winter vegetables, new york steak with mushrooms, mongolian beef kebabs with chile jam, beef stroganoff, beef boast braised in zinfandel, beef bourguignon, beef fried rice, golden pilaf, tasty tuna salad and cucumber chips, lemon grilled salmon, vietnamese beef and noodle soup, hungarian beef and potato soup, kenchin vegetable soup, beef mushroom barley soup, albondigas soup, cinnamon broiled chicken with raita, roast chicken dal, deep-fried chicken nuggets (tori no kara-age), sumac chicken with bread salad, spicy chicken wings, three cups chicken, chicken spread, sweet and sour chicken, fried chicken, chicken tikka, crescent samosas, basic (vegan) chocolate cupcake, grilled lime pound cake with raspberry sorbet, lemon-raspberry cake, chocolate cherry muffins, red velvet cake with cream cheese frosting, crabmeat with wasabi mayonnaise, naganegi stuffed with crabmeat, milk chocolate crêpe terrine, cheese manicotti, pie macaroni and cheese, seaweed and potato patties, tropical milkshakes, cherry cranberry shakes, mango margarita, americano cocktail, chai tea latte, breakfast cereal muffins, blueberry breakfast smoothie...dan banyak lainnya.

Website: http://resep-ok.blogspot.com/

http://sehat-enak.blogspot.com/

Kesehatan Milik Semua

Penyakit menular dari satwa liar, jenis penyakit dan pencegahannya, penyakit kelamin, penyakit kulit, penyakit mulut dan gigi, penyakit THT, tips kesehatan, dunia bayi, AIDS, Asma (Asthma), Cacar (Herpes), Darah Rendah (Hipotensi), Darah Tinggi (Hipertensi), Demam Berdarah Dengue (DBD), Demam Tifoid, Diabetes Mellitus (DM), Gagal Ginjal, Gagal Jantung, Gondongan (Mumps atau Parotitis), Hepatitis, Jantung Rematik (PJR), Kaki Gajah (Filariasis atau Elephantiasis), Kanker Kulit, Kanker Leher Rahim (Serviks), Kanker Paru-Paru, Kanker Payudara, Kolera (Cholera), Leukemia (Kanker Darah), Malaria, Meningitis, Penyakit-Penyakit di Musim Pancaroba, Pneumonia, Radang Usus Buntu (Appendicitis), Serangan Jantung , Abses Periapikal, Bau Mulut (Halitosis), Celah Bibir (Bibir Sumbing) & Celah gigi, Gangguan Kelenjar Ludah, Gangguan Lidah, Gigi Terjepit, Gigi yang Pernah Retak, Copot, Gingivitis (radang gusi), Infeksi Herpes Pada Mulut, Kanker & Pertumbuhan Lainnya, Karies Gigi (Kavitasi), Kelainan Pada Bibir, Mulut & Lidah, Kelainan Sendi Temporomandibuler, Malocclusion, Masalah Gigi Yang Darurat, Penyakit Pada Bibir, Periodontitis (piore), Pulpitis (radang pulpa gigi), Rahang Retak, Rahang Tidak Pada Tempatnya, Resesi Gusi, Sakit Gigi, Sariawan (Chanker Sores, Ulkus..., Setelah Perawatan Gigi, Trench Mouth (Infeksi Vincent..., Apakah amandel harus dioperasi?, Apakah saya terkena sinusitis..., Bagaimana Menghindari Alergi Debu?, Bagaimana Mengobati Sinusitis..., Bagaimana Mengobati Telinga Berair, Gangguan Pendengaran Pada Anak, Hidung mimisan, Hidung sering tersumbat dan..., Kanker Laring, Kanker Leher Kepala, Kanker Leher Metastatik, Kanker Nasofaring, Kenapa anak saya belum bisa bicara, Kenapa telinga bisa berjamur?, Kepala Pusing Berputar (Vertigo), Kotoran telinga, bagaimana..., Mendengkur atau Mengorok, Pengobatan Gangguan Keseimbangan, Penyebab Sinusitis, Serangga Masuk Telinga, Suara Serak, Telinga Berair Berulang Kali, Telinga sakit tiba-tiba, Tuli Akibat Bising di Zaman Modern, Tuli Mendadak, Tumor Tenggorok (Karsinoma..., AIDS (Acquired Immune Deficiency syndrome), Chlamydia, Gonorhoe (Kencing Nanah), HPV (Human Pappillomavirus), Jenis-Jenis Penyakit Kelamin, Penjabaran Singkat Penyakit Kelamin, Penyakit Kelamin Pada Wanita, Penyakit Menular Lewat Hubungan seks, Sifilis (Raja Singa), Sifilis dan Pengembangan Penyakit AIDS, Kandidiasis (Moniliasis), Eksim, Kadas / Kurap, Keratosis Pilaris, Gatal-gatal, Obat Tradisional untuk Penyakit Kulit, Penyakit Kulit dan Pencegahannya, Tinea Versikolor (panu), Dermatitis Seboreik, Herpes, Leptospirosis, Salmonellosis, Psitacosis, Toxoplasmosis, Cacing, Rabies, Tuberculosa (TBC), Hepatitis, Kegemaran Bayi, Inveksi Kulit akibat Virus, Tips-tips perawatan bayi, Grafik Berat badan Bayi, Bahasa Tubuh Bayi, Gejala dan Tindakan Pengobatan, Infeksi parasit pada kulit, Penyakit kulit menular, Tips Merawat Kesehatan Mata, Serba-Serbi Obat Warung, Pantangan Minum Susu, Ayo Diet Demi Sperma Sehat, Cerdas Dalam Memilih Air Minum, Seks Bagian 1: Seks 3x Sehari,,, Agar Asam Urat Tak Kambuh, Posisi Seks Paling Digilai Wanita, Bijak Memilih Obat Batuk, Tips Untuk Susah Tidur (Insomnia), Lelaki Multi Orgasme, Dambaan Wanita, Empat Saran agar Selalu Ereksi, Gairah Seks Wanita Menurun, Pria..., Konsumsi Lima Antioksida Eksogen, Posisi Bercinta Aman untuk..., Mengusir Cacing dengan Herbal, Ini Dia 7 Gangguan Saat Bercinta!, Sate Penyebab Kanker, Lima Sekawan Penghalau Lemak, Hindari Olahraga Berlebihan, Menjamin Ibadah Tetap Lancar, Pusat Seks Ada di Kepala Anda, Kiat Aman Mengonsumsi Suplemen..., 6 Titik Rangsang Mengejutkan, For Men: 8 Ways for A Healthy Life
Anemia (Kurang darah), Kelainan Sel Plasma (Diskrasia Sel Plasma, Gammopati Monoklonal), Leukemia, Hemofilia, Purpura Alergika, Trombositopenia, Sindroma Hemolitik-Uremik, Disfungsi Trombosit, Penyakit Hodgkin (Limfoma Hodgkin), Limfoma Burkitt, Mikosis Fungoides, Mieloma Multipel (multiple myeloma), Makroglobulinemia, Polisitemia Vera, Neutropenia, Limfositopenia, Kelainan Monosit, Eosinofilia, Limpa Pecah, Pembesaran Limpa, Biologi Darah, Transfusi Darah, Penyakit Jaringan Ikat Campuran, Persendian Charcot (Penyakit Sendi Neuropatik), Sindrom Carpal Tunnel, Ganglia, Penyakit Kienböck's, Sindrom bahu-tangan, Sindrom Cubital Tunnel,
Sindrom Radial Tunnel, Kelemahan Otot, Distrofi Otot Duchenne & Becker, Miopati Bawaan (Congenital Myopathies), Osteoartritis, Artritis Rematoid, Kanker Tulang Primer, Tumor Tulang Metastatik

Website: http://sehat-enak.blogspot.com/

Backlink Directory

Dapatkan 500 Backlink blog kami...

Direktori 103

Leap day hackathon for Google Gadgets, Maps, and more

Speaking the language of robots

First date with the Googlebot: Headers and compression

http://business-catalog.blogspot.com/

http://www.backlink4u.net/

http://resep-ok.blogspot.com/

http://sehat-enak.blogspot.com/

Link Directory

Backlink Directory

Lelah pasang iklan ?

500 Backlink langsung mengarah ke Blog Anda !

Ratusan Backlink permanen murah untuk blog

Blog Archive