Is it possible to extract data using XPath or regex?

[Wachynaky]

First of all, great job! I absolutely love it!
I noticed that you can extract both the Title and Description. Are there options to extract other fields? Is it possible to extract data using XPath or regex?

[janreges]

Hi @Wachynaky,

I have just implemented the ability to use XPath and RegEx in --extra-columns in commit cd6d55a. It will be released in the next version, or if you can, you can run the CLI version from the current main branch according to documentation.

XPath usage: Heading1=xpath://h1/text()
RegEx usage: ProductPrice=regexp:/Price:\s*?(\d+(?:\.\d{2})?)/i#1

[Wachynaky]

Hi @janreges works fine!

Extract date from meta <meta property="article:published_time" content="2025-02-12T13:02:18+00:00" />

./crawler \
  --url=https://www.mecagoenlos.com/Posicionamiento/analizando-mas-de-72.php \
  --extra-columns="cabecera=xpath://meta[@property=\"article:published_time\"]/@content" \
  --max-visited-urls=1 \
  --output=json 
Progress: 1/1     | 100% |>>>>>>>>>>>>>>>>>>>>>>>>>| 776 ms | https://www.mecagoenlos.com/Posicionamiento/analizando-mas-de-72.php

 "results": [
        {
            "url": "https:\/\/www.mecagoenlos.com\/Posicionamiento\/analizando-mas-de-72.php",
            "status": "200",
            "elapsedTime": 0.776,
            "size": 235393,
            "type": 1,
            "cacheTypeFlags": 7171,
            "cacheLifetime": 0,
            "extras": {
                "cabecera": "2025-02-12T13:02:18+00:00"
            }
        }
    ],

`