Skip to content
OSULZER.AT

Web crawler & bot development.

5.0 starsfrom 34 Google reviews

Read data from websites automatically, prepare it and make it available - from Vienna, for clients across Austria.

See services

Services

Data that works for you.

I build crawlers and bots that read information from websites automatically, prepare it and deliver it exactly where you need it.

Key features

  • Scraping

    Important information is read from websites automatically and stored in the form you need - for example as a spreadsheet or directly in a database.

  • Processing

    Statistical analyses, data preparation, filtering or delivery via API - exactly as you need it.

  • Automation

    Bots carry out recurring work tasks automatically - saving you time and nerves.

  • Specialized search functions

    Do you regularly search marketplaces for real estate or products and have very specific requirements? Save yourself the work and let the listings be read and narrowed down automatically.

  • Competitor analysis

    You are notified automatically when competitors change prices or products or add new ones.

  • Data migration

    Bots also help when moving static websites into a CMS: the pages are read automatically and then imported into the new system. Especially interesting for large, aging projects without a database connection or export function.

How a crawler reads

Crawlers extract information from websites fully automatically, and with AI support if you like.

A crawler visits a website just like a visitor. It reads the text line by line, follows every link and recognizes what matters: product names, prices, addresses or dates.

It plays by the rules of the website: whatever the robots.txt blocks, it leaves out, and it sends requests so gently that the server never breaks a sweat.

With modern technology, such as a real browser in the background and natural pauses, it behaves like a person at a screen and is hard to tell apart from a normal visitor.

Whatever it finds, it stores in an orderly way, for example as a spreadsheet or straight into your database. The next day it comes back and reports only what has changed.

The essentials at a glance

How a crawler works.

Do you want data from the internet read automatically and prepared in a specific way? A crawler visits websites, “scrapes” the information you want and collects it for you - for example in a database.

01

Visit the website

The crawler opens the pages you want automatically - as often and as regularly as you need.

02

Extract the data

From each page it “scrapes” exactly the information you care about: prices, titles, descriptions, contact details …

03

Prepare & filter

The data is cleaned, checked, filtered and analyzed - for unstructured pages, with the help of AI.

04

Deliver

The result ends up where you work with it:

  • CSV / Excel
  • Database
  • API
  • Web interface

More than just reading

Beyond passive visits, bots can also interact with websites if you like - and take recurring tasks on the internet off your plate.

  • Fill in forms
  • Send emails
  • Simulate mouse movements
  • and much more

Built for you

Specialized & scalable.

  • Tailored & resource-friendlyYour crawler is tailored to your use case and uses as few resources as possible.
  • Multiple instances, separate proxiesIf a single crawler is not enough, several instances get to work - through different proxy servers if you like.
  • ExtensibleEvery crawler can be extended with more features - for example, to read additional websites.
  • Data in a web interfaceIf you like, I present the collected data in a practical web interface - built around your requirements.
One crawler, many routesOne crawler starts three instances, and each one fetches the websites through its own proxy server.

Use cases

What a bot is good for.

Are you working on a project that eats up time with recurring tasks? Pick a use case and see how a crawler helps.

Keep an eye on your competitors’ prices

Do you sell products online and want to offer your customers the best prices? Specialized bots read your competitors’ prices automatically and tell you about important changes.

Price comparison · dailyExample
ProductYour priceCompetitor
Coffee grinder Classic€89.00€92.50
Milk frother€59.00Price changed€64.00 €54.90
Espresso cups, set of 6€34.90€34.90
Descaler 1 l€12.90€13.50

Notification to you: 1 price has changed.

Current & proven

Technologies used.

Every project is unique and has different requirements - depending on the task, other technologies come into play. We can talk about the details in a personal conversation.

  • Fetching pages

    From a quick single request to a bot that works its way through entire websites - in Python, platform-independent.

    • Pythonruns on Windows, Linux or macOS
    • httpxasynchronous with HTTP/2 - fast for many requests
    • Scrapyframework for large, robust crawlers
  • Dynamic pages & interaction

    For websites that load content via JavaScript or only show it after logins, clicks and input.

    • Playwrightcontrols Chromium, Firefox and WebKit - waits for content automatically
    • scrapy-playwrighta browser only where a page really needs one
  • Parsing HTML

    Pull exactly the information you need out of a page’s source code.

    • selectolaxvery fast HTML parser
    • parselprecise with CSS and XPath selectors
  • AI-assisted extraction

    A language model also reads unstructured pages and delivers clean, structured data - robust even when the layout changes.

    • Local LLMsstructured data as JSON, no cloud
    • Crawl4AIopen-source crawler that prepares pages for AI
  • Data & output

    Validated records, in exactly the format you continue working with.

    • Pydanticchecks every record for completeness and format
    • PostgreSQL & FastAPIdatabase or custom API - or CSV/Excel instead
  • Operations

    Crawlers run on a schedule, in isolation, and are easy to update.

    • Dockercontainers for easy updates
    • Schedulingcron or Prefect - regular and monitored

Operations & maintenance

Up and running - and staying that way.

A crawler needs a reliable place to work and an occasional check-up. You get both from me, from a single source.

Technical requirements

Hosting

For your crawler to work flawlessly, a dedicated server is recommended.

  • Depending on scope, a virtual or a dedicated server
  • I am happy to recommend suitable hosting providers and plans
  • If you like, I host your crawler myself

Keeping everything running

Maintenance

To keep your crawler reliable, I recommend regular maintenance.

  • Checking that the crawler runs as intended and collects the right data
  • Adjustments when a data source gets a redesign and presents information differently
  • If you prefer, only once the crawler stops working properly

How it works

How we work together.

From the first idea to a running crawler - with one point of contact for development, hosting and maintenance.

  1. Step 1: Free initial consultation

    We talk about your data sources, the information you want and what you need it for - free of charge and without obligation.

  2. Step 2: Analyze the source

    I take a close look at the websites: how is the data structured, is it loaded via JavaScript, is a login required?

  3. Step 3: Prototype

    A first crawler extracts sample data - so you can see early on whether the results match what you have in mind.

  4. Step 4: Development

    The crawler is tailored to your requirements, built to be resource-friendly, and the output is prepared the way you want it.

  5. Step 5: Hosting & launch

    The crawler moves to a dedicated or virtual server. I am happy to recommend suitable providers - or I host it myself.

  6. Step 6: Maintenance

    Regular checks and adjustments when a source changes its design - or, if you prefer, only once it becomes necessary.

Your web designer: Oliver Sulzer

Let's get to work!

As an experienced freelance developer, I build crawlers and bots that take recurring work off your hands: they extract data, prepare it and let you know when something changes. Every crawler is tailored to your use case - and after launch I remain your point of contact when a source changes. I am based in Vienna and support clients across Austria.
years of experience
15+
completed projects
50+
websites looked after
50+
usually within one business day
Response
Oliver SulzerWeb designer & developer from Vienna, more than 15 years of experience

Guides on this topic

From the blog.

Guides and solutions from real projects - related to this service, to read up on or do yourself.

All articlesRead all 44
articles

Good to know

Web crawlers & bots.

Login-protected pages, JavaScript, scaling, servers and maintenance: here are the most common questions about crawlers and bots.

Possibilities

Can a crawler also read pages that are only visible after a login?

Yes. With Playwright, the bot controls a real browser, logs in, clicks and fills in forms - and then reads the content that only becomes visible after logging in. For large projects, I combine this with the Scrapy framework.

Does this also work for websites built with JavaScript?

Yes. Such websites are a challenge because content is often loaded dynamically. For that I use Playwright: it controls a real browser and automatically waits until the content has loaded. Because a browser needs more resources, it is only used where a page really requires it.

Can AI help with extracting data?

Yes. For unstructured pages - such as product descriptions or contact details buried in running text - a language model reads the content and returns it as clean, structured data. This stays robust even when a page’s layout changes. If you like, the model runs locally on your own hardware, with no cloud at all.

What if a single crawler is not enough?

Then the crawler works in several instances - through different proxy servers if you like. Every crawler can also be extended with more features, for example additional websites or a web interface for the collected data.

Technology & operations

What technical requirements do I need?

For flawless operation, a dedicated server is recommended - depending on scope, a virtual or a dedicated server. I am happy to recommend suitable hosting providers and plans. If you like, I can also host the crawler myself.

Does the crawler run on Windows, Linux or macOS?

Yes, on all three. Because I build crawlers in Python, they are platform-independent and can run on Windows, Linux or, if you like, macOS.

Does a crawler need maintenance?

I recommend regular maintenance. I check that the crawler runs reliably and collects the right data. Specialized crawlers in particular need adjustments when a website that serves as a data source gets a redesign and presents information differently. If you prefer, I only take care of it once the crawler stops working properly.

What my clients say.

34 reviews, all with 5 stars - from shops, agencies, start-ups and universities.

„Mehr als nur 5 Sterne verdient“

Wir sind Betroffene von der aktuellen Google Fonts Abmahnwelle. Unsere alte Website entsprach nicht der DSGVO und eine Anpassung war technisch nicht mehr machbar. Es musste also ganz rasch kompetente Hilfe her, quasi eine „Internetfeuerwehr“ :-) Oliver hat die Dringlichkeit erkannt und innerhalb kürzester Zeit unsere Seite - nicht nur - nachgebaut, sondern gleichzeitig modernisiert, etwas peppiger gemacht und - das ist am wichtigsten - DSGVO-konform umgesetzt. Oliver hat sich als zuverlässiger und termintreuer Partner mehr als nur 5 Sterne verdient. Weiter so Oliver, wir bleiben Dir sicher auch in Zukunft treu! Beste Grüße von Geri und Manuela

Lack & LederReview on Google
Google

5.0

34 reviews on Google

„Top Umsetzung aller Vorstellungen - unkompliziert und genau!! Danke dafür :)“

Doris P.

„alles easy, schnell und zuverlässig.“

Isabella fteval

„Guter und schneller Support. Sehr zufriedenstellend!“

Bertram H.

Trusted by companies, agencies & educational institutions

From sole proprietors to universities - websites, online shops and WordPress support.

  • Hochschule Campus Wien
  • PHH Rechtsanwälte
  • Pfeffer & Partner
  • RAVN

„Freundlicher Kontakt, professionelle Umsetzung und direkter telefonischer Kontakt!“

Jürgen K.
All reviewsRead all 34
reviews

Reviews shown in their original German.

New project?

Contact.

Tell me briefly about your project - I'll get back to you personally.

Your data is used solely to handle your inquiry. More in the privacy policy.