What's New

New updates and improvements to SearchApi

February 2024: Update

New
This month, we’ve rolled out several updates that we think you’ll find valuable: a new integration, significant engine and performance improvements, and a fresh search history feature.

🌟 ​Bing Search Engine​


We're excited to release Bing Search API offering organic results, ads, local results, related questions, and related searches. Similarly to Google, it supports precise location-based targeting using our ​Locations API. If that's not enough, you can also use latitude and longitude coordinates for coordinate-level precision. 

To ease integration, JSON keys align closely with those of the Google Search API. Perfect for building tools across different search engines.

🚀 Major Performance Upgrade


We've optimized our API, focusing on speed and stability. Key steps included revamping our proxy network on January 4th and boosting server capacity while fine-tuning our proxy network on a per-engine basis on January 12th, thanks to December’s backend efforts on the admin side.

As a result, SearchApi average response time is already less than 3 seconds. We anticipate improving it further to bring it below the 2-second threshold.

🔎 Search History Page


A new feature to review recent searches with the ability to filter by the search engine. We hope it will be really useful in debugging automated queries.

🔧 Engine Improvements:


Google Search API:

  • Added a new major parsing field: Sports Results. This includes three different types of sports results: 
    • Athlete Stats: This type can be triggered when searching for player statistics.
    • Team Sports: This type can be triggered when looking for games in basketball, football, or other team sports.
    • Single Sports: This type can be triggered when searching for golf, F1, tennis, or other individual sports games.



Google Shopping API:

  • Fixed product link and added `prds` value when there is no `product_id`.
  • Fixed documentation num parameter, as Google Shopping does not support the num=10 parameter and returns 60 results instead.

Google Product API:

  • Fixed an issue where the selected Product didn't return the main image of it.

Google Scholar API:

  • Fixed issues setting `num=20`.

Google Lens API:

  • Added mobile support as Google Lens returns larger images on it.
  • Added exact price and currency to visual_matches.
  • Added price, and stock information to exact_matches. Fixed source.
  • Fixed HTML. Now HTML shows an image, which was searched for as well.
  • Added ​country​ parameter. It allows making searches in a specific country to influence shopping results and the accuracy of place identification. 

Google News API:

  • Fixed top stories extraction.

Google Trends API:

  • Google Trends API is fully functional again.

Youtube Transcripts API:

  • API now returns all available languages even if the transcripts are not available for selected languages. Check ​documentation for an API example.

Amazon Product API:

  • Added bundle items & bundle carousel.
  • Improved variants extraction.
  • Improved buybox extraction.
  • Improved compare_with_similar table parsing.

🔗 New Integrations:


  • ​search_with_lepton​ - A dynamic, LLM-supported conversational search tool with an intuitive interface and shareable, cached results for efficient information retrieval.

Other improvements:


  • Transitioned to Stripe Embedded Checkout.
  • Upgraded to Ruby 3.3 in our production environment.

⏭️ What are we working on next?


  • Google Maps API
  • YouTube Channel API
  • Amazon Reviews API
  • Consider a CORS policy update for easier API experimentation.

Stay tuned for more updates! As always, we're committed to providing you with the best experience.

January 2024: Update

New
Happy New Year! We're excited to start 2024 by improving your experience with SearchApi. This month, we've worked hard to improve our engines, especially Google. We believe these improvements will make your work easier and help you discover new things more efficiently.

🔧 Engine Improvements:


Google:

  • Added "Discussions and Forums". The element shows forums and their topic based on your query.
  • Implemented a new case of sitelinks - list - for organic results.
  • Added new element "From sources across the web" that shows sites that cover a specific topic in this grid list view.
  • Fixed local_ads and ads on mobile and desktop.
  • Fixed inline_images.
  • Fixed top_stories.
  • Added weather_results on mobile for up-to-date weather information.
  • Fixed the snippet in organic_results. Google started testing longer snippets, which might be related. 

Google News:

  • Fixed top_stories.

Google Shopping:

  • Fixed an issue when filters didn't return any text.
  • Added Google Shopping pagination.

Google Product:

  • Fixed an issue where typical_prices didn't return any results.

Google Lens:

  • New fields are being parsed - videos, related_searches, and exact_matches.

👨🏻‍💻 Product Changes


  • Enabled Link, Apple Pay & Cash App in the checkout. 
  • Added a status page to display uptime and performance metrics.
  • Released the first version of the ​Press Page. It provides marketing material for the affiliate program.

⏭️ What are we working on next?


  • Testing Bing API & Google Maps API that will be released this month.
  • Work in progress for API usage stats and search history features, initially for the admin side, with plans to extend to user accounts.
  • Even better API response time. We are in the process of optimizing our proxy pool. Thanks to the work done on the admin side. 
  • More integrations. 

Stay tuned for more updates! As always, we're committed to providing you with the best experience. Got feedback or questions? Just hit reply.

December 2023: Update

New
We're excited to bring you our latest updates for this month, including two new integrations, a major engine update, and an affiliate program.

🌟 New Engines


This is a single reason our update is late this month. We wanted to polish and release it before Christmas. 🎄 

Use the Amazon Product API to scrape product details, prices, delivery information, ratings, reviews, and more. It supports 22 Amazon domains and retrieves product information by ASIN.

We made sure to parse as much information as possible on the page and we think this is one the most comprehensive parsers on the market right now. We will still polish it during the week, so expect some minor changes. 

Customizing the delivery location country and zipcode is one of the key features we will be exploring at the start of 2024 and expanding the Amazon support further. 


Use the Google Autocomplete API to scrape real-time search autocomplete suggestions, their relevancy, popularity, and type. It supports nine different clients, including Chrome, Firefox, and Safari, each offering unique autocomplete results. Google Autocomplete API comes with a customizable cursor position, allowing different variations of suggestions.

💰 Affiliate Program (30% revenue share)


Promote SearchApi and earn a 30% revenue share for every new customer you refer.

We partnered with Rewardful as our affiliate software. We thoroughly tested our system, so it correctly tracks all people registering on the platform, even through the Google and Github logins. Learn more about the program on the ​affiliate program page​.

After signing up you get immediate access to your personal affiliate link and promotion materials, which you can use to recommend the SearchApi. Building integrations, self-hosted tools, and writing or organizing webinars are just a few ways to get started. Head over to the ​affiliate page​ for more ideas.

🔧 Engine Improvements:


Google

  • The new "Things to Do" block displays the most popular places to visit in the selected location.




Google Scholar

  • Improved snippets to return full text hidden under "show more" in DOI or full article title searches.




🔗 Integrations


  • ​SerpBear​ - a simple, self-hostable, open-source SERP position tracking app. It allows you to track your website's keyword positions in Google and get notified of their positions.
  • ​Langchain​ (Python) - when integrating SearchApi into the Langflow, we noticed some issues, mainly with imports. This ​PR​ fixed the issues and has been released in ​the v0.0.346​ version.
  • ​HayStack​ - LLM orchestration framework to build customizable, production-ready LLM applications. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search, or conversational agent chatbots.

Other changes

  • Improved accessibility on the landing and documentation pages.
  • Improved loading speed for the documentation pages.
  • SEO improvements: markup & semantic tags.


⏭️ What are we working on next?

  • Development of Bing Search API and Google Maps API.
  • A status page to display our uptime and performance metrics.
  • Work in progress for API usage stats and search history features, initially for the admin side, with plans to extend to user accounts.
  • More integrations are underway. 

Stay tuned for more updates! As always, we're committed to providing you with the best experience.

November 2023 Update

New
As another month wraps up, we're eager to share our latest strides in enhancing your development workflow.

🛠️ Developer Experience


  • Integration Examples. Expanded support for 13 programming languages, enhancing our library support to facilitate seamless integration with SearchApi. We've introduced intuitive icons for quick actions, enabling you to copy code snippets or open API examples in a new tab more efficiently.
​​



  • API Key Management
    • Multiple API Keys. ​The new Account API keys page​ allows the creation of distinct API keys for different environments or individual team members, reinforcing your operational security posture.
    • Zero downtime rotation. Transition between old and new API keys without service interruption, ensuring continuous operation.
    • Default API key. The default API key is used sitewide across all integration examples, so you can test the API faster.
    • The default API key is now hidden in the dashboard. 

🔧 Engine Improvements


Google
  • New "Salary Estimates" block providing valuable compensation data.



  • New "About this result" block offering contextual information about search results.



  • "People also search for" block is now parsed as `related_searches` on mobile.
  • Added pagination for Google searches, News, and Jobs.

Google Lens
  • Now supports query parameters in image URLs, correcting previously encountered encoding issues.

👨‍💻 DevOps


  • Resolved 504 HTTP response timeouts by adjusting the CloudFront to accommodate longer wait times for AWS ALB responses and established CloudWatch alerts for proactive monitoring.
  • Upgraded our infrastructure to utilize Rails 7.1.1 & Ruby 3.2.2 for improved performance and stability.

🔗 New Integrations


  • Flowise Integration. A new addition to facilitate low-code/no-code development for LLM applications.
  • Pipedream Integration. Enables connection to over 1000+ APIs with options for both coders and non-coders. Detailed information is available on our ​SearchApi Pipedream page​.

⏭️ What's coming next?


  • Amazon Product API
  • Google Maps API
  • TikTok API
  • More integrations

Your engagement is crucial for our continuous improvement. We encourage you to test these updates and look forward to your valuable feedback. Thank you for being an integral part of the SearchApi developer community.

October 2023 Update

New
We're thrilled to update you on some of the newest enhancements and give you a sneak peek into what’s coming next.

🦜🔗 LangChain Integration

Large language models (LLMs) are emerging as a transformative technology, enabling developers to build applications that they previously could not. However, using these LLMs in isolation is often insufficient for creating a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge.

By embedding the SearchAPI, the Large Language Model can respond by querying real-time data to answer questions about current events. Its usage is particularly interesting with the "Self-ask with search" agent. This agent is equivalent to the original ​self-ask with search paper​, where a Google search API was provided as the tool.

Another common user case is summarizing long documents that naturally run into context window limitations. Here is an example ​using SearchApi to load a lengthy YouTube video transcript​, then summarize and distill it with a refine chain.

SearchApi is now natively available as part of ​langchainjs​ and ​langchain​ libraries.

For a quick overview, check out this tweet on SearchApi integration in LangChainJS.

Here is the official SearchApi documentation on its usage in Python and Javascript:
  •  Python examples​
  •  JavaScript/TypeScript examples​​

🌟 Freshly Released Engines:

 
✔️  Google Lens API​ - shipped in less than 24 hours based on ​@levelsio tweet on Twitter​. Useful as a tool in ​affiliate marketing​ for fetching product sources, links, prices, and stock information.
  ✔️  YouTube Transcripts API​
  ✔️  Amazon Search API​

🔧 Engine Improvements:

Google
  •  Added the ability to select a custom date range.
  •  Fixed `inline_videos` parsing on mobile
  •  Added `source` to desktop & mobile organic results
  •  Fixed `detected_location` in `search_information`
  •  Fixed `date` and `snippet` identification in organic results

⏭️ What's coming next?

  •  Google Maps engine
  •  Amazon Product engine
  •  Integration examples in multiple languages and libraries
  •  LangChain integration (Ruby)

Keep an eye out for further updates! We're dedicated to ensuring you have the best experience. 

September 2023 update

New
During the last month, we added 6 new engines, made significant improvements to the API response time, and made sure it is highly scalable.

🌟 Freshly Released Engines

 
✔️  Google Product Offers​
  ✔️ ​ Google Product Reviews​
  ✔️  Google Product Specifications​
  ✔️  Google Scholar​
  ✔️  Google Scholar Cite​
  ✔️  Google Scholar Profiles​

🔧 Engine Improvements

Google
Introduced an `organic_results` type Answer Box field for desktop devices.

Google Play Store
Resolved an issue with the next_page_token. When searching for a specific application, the next_page_token will now reliably return the second page of Similar Products, eliminating previous errors.

Google Trends
Added the ability to select a custom date range.

🚀 Performance Improvements

  • Optimized all key database calls. (Thanks to NewRelic)
  • Refactored API usage tracking. This will allow us to add usage charts.
  • Identified a blocking operation in our API and made it async. Saved `~70ms` on API response time.
  • Performed load testing. We managed to solve all the application and infrastructure issues.
  • Experimented with Rust for HTML parsing. It allowed us to improve the parsing time by up to 10x. This is still very much WIP.

⏭️ What's coming next?

  •  Google Maps engine
  •  Amazon Search engine
  •  YouTube Transcripts engine 
  •  Integration examples in multiple languages and libraries 
  • Langchain integration (Python) 

Stay tuned for more updates! As always, we're committed to providing you with the best experience.

August 2023 Update

New
The past month has been another period of high productivity for our team, with considerable upgrades predominantly made to our Google engines. Keep an eye out for more refinements as we persist in adapting and broadening our services in response to your valuable feedback and requirements.

🆕 New Features Released

✔️ Released Google Jobs engine
✔️ Released Google Videos engine
✔️ Released Google Images engine
✔️ Released Google Shopping engine
✔️ Released Google Product engine

⚡ Performance & DevOps

• Improved parsing time up to 4x
• Improved performance for all queries utilizing location geo-targeting. We are able to achieve geo-targetting consistently in 10ms-20ms (took up to 1s previously).
• Integrated CDN with 450+ Points of Presence and 13 regional edge caches in 90+ cities across 49 countries
• Zero-Downtime deployment (Blue-green)
• Integrated Continous Delivery process into our team workflow
• Upgraded infrastructure components to the newest versions available.
• Integrated NewRelic & connected to AWS for enhanced performance monitoring

⏭️ What's coming next?

The main focus is on expanding our selection of available engines and increasing the performance even more. The goal is to have the average response time below 2 seconds consistently. We are happy that a significant part of Google searches are below 1 second already.

• Google Product Offers engine
• Google Product Reviews engine
• Google Product Specifications engine
• Google Maps engine
• Google Scholar engine
• Integration examples in multiple languages and libraries
• More performance improvements

We appreciate you choosing our services. We're working hard to make things better for you. If you need help or have questions, just let us know. We're always here. Thanks for your support.

July 2023 Update

New
Our team has had another highly productive month, with significant updates rolling out primarily on our Google Search API. One of the key enhancements is the improved support for mobile and tablet devices across numerous fields. This marks a major leap in our commitment to delivering a seamless and user-friendly search experience on all platforms, making it more convenient and efficient for users to find the information they need. Stay tuned for further improvements as we continue to refine and expand our services based on your feedback and needs.

🆕 New Features Released

✔️ Released Google News engine
✔️ Released Youtube engine

Significant Google Search Engine Update

We're excited to announce that we've significantly expanded our parsing capabilities within the Google Search Engine. The update will incorporate several new fields, mobile and tablet devices support.

New fields in Google Search Engine:

Jobs
Local Map
Inline Tweets
Inline Recipes
Inline Images

Mobile and tablet is now supported for given fields:

• Inline Videos
• Jobs
• Top Stories
• Inline Tweets
• Ads
• Local Ads

Additional Updates

In order to further enhance your experience, we have:

• Fixed curl examples
• Changed product_info key in shopping_ads to extensions
• Changed documentation API key description
• In our continuous effort to enhance user experience, we have revised the titles of our API examples

⏭️ What's coming next?

• Google Jobs engine
• Google Maps engine
• Google Videos engine
• Google Images engine
• Google Scholar engine
• Google Shopping engine
• Google Search engine organic results tablet and mobile devices support


Your choice to use our services means a lot to us. We're dedicated to making your experience better. If you need help or have questions, please reach out. We're always here to help. Thanks again for your support.



June 2023 Update

New
We have completed another productive month with an emphasis on improving and broadening our Google Search Engine capabilities. Our team has added parsers for more than 100 distinct page blocks, reflecting our ongoing commitment to delivering unparalleled data quality and precision.

🆕 New Features Released

✔️ Released Google Trends engine
✔️ Released Google Trends Autocomplete engine
✔️ Released Google Locations API engine, which enables Google searches with high-precision coordinates across 195 supported countries.

Significant Google Engine Update

We've broadened our parsing scope in the Google engine to include the following fields:

Ads;
Local Ads;
Shopping Ads;
Inline Videos;
Inline Shopping;
Inline Recipes;
Related Questions;
Local Results;
Top Stories;
• Knowledge Graph (with emphasis on Business, Shopping, and Jobs entities).

Additional Updates

In order to further enhance your experience, we have:

• Integrated Intercom live chat support to offer instant assistance.
• Automated embedding of the api_key in all documentation examples when you are signed in for ease of use.
• Resolved issues with product_highlights extraction in Google Play Store.

We thank you for choosing our services and we're committed to continuously improving and providing the best experience for you. If you have any questions or need further assistance, please do not hesitate to reach out.

May 2023 Update

New
🚀 Official Launch Announcement

Though we're a new face in the SaaS world, we're far from inexperienced. For years, we've been providing advanced web scraping solutions with high-performance APIs for our private clients. Now, we're thrilled to make our services available to everyone.

Our mission is simple: to make public data accessible to everyone, with precision, quality, and rich, extensive information at the core of everything we do, while ensuring our APIs perform efficiently and reliably.

Join us on this journey, and as a new startup, we pledge to deliver unparalleled technical support and direct communication with our engineers. This commitment, coupled with our focus on API performance, sets us apart and remains our number one priority and unique value proposition.

🆕 New Features Released

✔️ Google Play Store engine.
✔️ Google Play Product engine.
✔️ Initial version of Google search engine. We parse organic_results and related_queries on the desktop.
✔️ 100 free searches - newly registered users can test our platform with 100 complimentary searches.

⏭️ What's coming next?

• Significant Google search engine improvements. This includes precise search by geolocation and a bunch of new blocks rolling over the next weeks.
• Trends Search API
• Youtube Search API
• Infrastructure upgrades - we're enhancing our infrastructure and conducting stress tests to accommodate even larger scale projects.

Join us today and experience the difference of our advanced web scraping solutions. Welcome aboard!