Transform any website into an API with Python / BeautifulSoup

Transform any website into an API with Python BeautifulSoup

I love APIs. They’re an amazing source of data for all kinds of creative projects.

But what if a website doesn’t provide an official API?

Can you still collect some data programmatically on a regular basis?

If you want to transform (almost) any public-facing website into an API, you can leverage the power of Python, using one of the scraping packages developed by the community.

If you need to interact with the page (fill out a form for instance), you’ll use Selenium. For the purpose of this article, we’ll use a simpler package, which works perfectly fine for most use cases: BeautifulSoup (bs4).

The first requirement is of course to have Python installed on your machine and pick your favourite IDE (I use PyCharm). If you’re new to Python, you can read my detailed tutorial explaining how to install it on your machine.

I’ll be using a Jupyter Notebook on Kaggle, which I will share at the end of this article. That’s a great way to run Python code in the cloud, step by step.

In the first cell of my notebook, I’m installing the bs4 package, in a persistent way, in my Kaggle workspace. This will enable me to have bs4 available for further sessions using the same notebook (otherwise, you have to re-install bs4 to use your script).

!pip install bs4 --target=/kaggle/working/mysitepackages

I run the code in the cell, then I can replace it by the import command and run the cell again. If there’s no error message, we’re good to go.

We just have to import another package (we can do it in the same cell & re-run it), which doesn’t require any further installation: requests. That’s the package you will always use to connect either to a public API or to call your own API using bs4 or another method. For the purpose of this demo, we’ll also import a built-in package to display the current date.

from bs4 import BeautifulSoup
import requests
from datetime import date

How to get the data for the 10 Best Selling fiction books on Amazon?

Let’s say that we want to get the Best Selling fiction books on Kindle, which you can find on

Let’s first connect to the web page.

#declaring the URL
url = ""
# declare browser headers
headers = {'Accept-Language': "fr,en-GB;q=0.9,en-US;q=0.8,en;q=0.7",
           'User-Agent': "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36"}

#getting the full content of the page
page = requests.get(url, headers = headers)

#parsing the HTML
soup = BeautifulSoup(page.content, "html.parser")

In the next cell, we can print our “soup” (= the html content of the page)


Here’s the kind of content you can expect. A slightly indigestible soup 😉

amazon soup

Now comes the fun part. We have to figure out how to get the data for those books.

Amazon Fiction Best Sellers
Amazon Fiction Best Sellers – Feb 2023

Head over to your Chrome browser and INSPECT a relevant element on the page.

That’s the kind of code you’ll see in the Elements tab of your Developer Tools.

Inspect the CSS of your page

You might have noticed the id=”rank1″ of the DIV containing the information.
Let’s try to access it.

digger api
number_one_book = soup.find(id="rank1")

Let’s print “number_one_track” in the next cell.


We get the code for all the info about the #1 Best Selling fiction book on Amazon Kindle.

It’s still pretty indigestible… 🤢

Screenshot 2023 02 01 at 12.52.15

Now we have to locate the TITLE and the AUTHOR in the code. We’ll also find the PUBLISHER.

<div class="kc-rank-card-title">
        It Ends with Us
<div class="kc-rank-card-author" title="Colleen Hoover">
            by Colleen Hoover
<div class="kc-rank-card-publisher" title="Atria Books">
            PUBLISHER: Atria Books

That was easy.

So let’s write some code to focus on those 3 pieces of information.

number_one_book_title = soup.find(id="rank1").find(class_="kc-rank-card-title")

You’ve noticed how I chained two .find() commands, to find the rank1 div id and the kc-rank-card-title class. Take note of the underscore after “class” (class_=”target-class”).

Let’s print this out, narrowing down to the text (.text), removing any white spaces (.strip( )).

#prints out: 
It Ends with Us

Perfect. Now let’s do the same for the author & the publisher and compute today’s date in a readable format. We will also adapt the code for the title extraction to get the clean text string.

number_one_book_title = soup.find(id="rank1").find(class_="kc-rank-card-title").text.strip()
number_one_book_author = soup.find(id="rank1").find(class_="kc-rank-card-author").text.strip()
number_one_book_publisher = soup.find(id="rank1").find(class_="kc-rank-card-publisher").text.strip()
today =
# dd/mm/YY
d1 = today.strftime("%d/%m/%Y")

We can also get the image of the cover and of course the link to the ebook, which I will build using the slug (the end of the URL) found in the code.

number_one_book_cover = soup.find(id="rank1").find(class_="kc-cover-link").find("img").get("src")

You have noticed how I chained multiple commands (find, find, find, get).

This will give me:

number_one_book_slug = soup.find(id="rank1").find(class_="kc-cover-link").get("href")
number_one_book_url = "" + number_one_book_slug

The full URL will be:

Now we can build a full output by concatenating the data.

We’ll wrap the info in HTML to show the image.

widget = f'<div style = "border: solid 1px black; text-align: center; padding: 10px"><p><span>{d1} - Number One Fiction Book On Amazon</span><br><b>{number_one_book_title}</b><br><span>{number_one_book_author}</span><br><span>{number_one_book_publisher}</span><br><br><span><a href={number_one_book_url} style="padding: 0.5em 1em; background-color: blue; color: white; text-decoration: none;">Buy the book</a></span><br></p><a href="{number_one_book_url}"><img src={number_one_book_cover} alt="{number_one_book_title}" width=100px></a></div>'

👉 If I print out the result of this widget (print(widget)) and copy-paste it into a HTML block on this WordPress site, here’s the result:

01/02/2023 – Number One Fiction Book On Amazon
It Ends with Us
by Colleen Hoover
PUBLISHER: Atria Books

Buy the book

It Ends with Us

Now let’s imagine that we want the Top 10 of the best selling fiction books.

We won’t create a widget for the list (we could) but we’ll print out the results, along with the links to the books.

We have noticed that each book card has a rank{number} class. So, if we want to get the 10 best selling books, we can proceed this way:

print(f"10 Best Selling fiction books on Amazon - date: {d1}")

rank = 1
while rank <= 10:
    book_title = soup.find(id=f"rank{rank}").find(class_="kc-rank-card-title").text.strip()
    book_author = soup.find(id=f"rank{rank}").find(class_="kc-rank-card-author").text.strip()
    book_publisher = soup.find(id=f"rank{rank}").find(class_="kc-rank-card-publisher").text.strip()
    book_slug = soup.find(id=f"rank{rank}").find(class_="kc-cover-link").get("href")
    book_url = "" + book_slug
    print(f"Ranked {rank}: {book_title} - {book_author} | {book_publisher}. Link: {book_url}")
    rank += 1

👉 Note: I used f strings (f”{}”) to mix text and variables: f”text{variable}”

If I run this cell, I get the following output:

10 Best Selling fiction books on Amazon - date: 01/02/2023

Ranked 1: It Ends with Us - by Colleen Hoover | PUBLISHER: Atria Books. Link:
Ranked 2: Lessons in Chemistry - by Bonnie Garmus | PUBLISHER: Doubleday. Link:
Ranked 3: It Starts with Us - by Colleen Hoover | PUBLISHER: Atria Books. Link:
Ranked 4: Verity - by Colleen Hoover | PUBLISHER: Hoover Ink, Inc.. Link:
Ranked 5: Remarkably Bright Creatures - by Shelby Van Pelt | PUBLISHER: HarperAudio. Link:
Ranked 6: Tomorrow, and Tomorrow, and Tomorrow - by Gabrielle Zevin | PUBLISHER: Knopf. Link:
Ranked 7: The Housemaid - by Freida McFadden | PUBLISHER: Bookouture. Link:
Ranked 8: Demon Copperhead - by Barbara Kingsolver | PUBLISHER: Harper. Link:
Ranked 9: The House at the End of the World - by Dean Koontz | PUBLISHER: Brilliance Audio. Link:
Ranked 10: Reminders of Him - by Colleen Hoover | PUBLISHER: Montlake. Link:

Mixing this technique with the tutorial I wrote about connecting Python to Google Sheets, you can store the data in a table for further processing.

👉 As promised, here’s the link to the public Jupyter notebook on Kaggle.

Have fun building your own personal APIs with Python and BeautifulSoup!

🚀 Subscribe to my weekly newsletter packed with tips & tricks around AI, SEO, coding and smart automations

☕️ If you found this piece helpful, you can buy me coffee