ratt(5)

NAME

ratt - lua config files syntax

SYNOPSIS

ratt reads config files int the current directory, then searches in ~/.config/ratt/*.lua and then searches the embedded configs.

SYNTAX

Config files are lua scripts. There are some defined attributes that ratt will interpret and use to extract data from the website and build the rss/atom feed.

The main function is ratt.add(), which takes two arguments. The first is the regex for matching the website url and the second is the selectors table.

regex
    Config files are matched with the url provided and the regex in the config file. The regular expression has the syntax of go regexp described at https://golang.org/s/re2syntax.

    regex: https://www.youtube.com/results.*

  CSS SELECTORS ATTRIBUTES

feed and item attributes can be either css selectors (strings) or lua functions.

selectors
    is a whole object that define how the data will be extracted. selectors object has the following attributes: httpsettings, feed, item, nextpage, nextpageattr, nextpagecount, sort

httpsettings
    (optional) describes the http request details, which ratt will send to obtain the website data

Every css selector has an accompanying attribute with the suffix 'attr'. If this attribute isn't empty, ratt will not get the data of the selected element, but the data from the element's attribute. e.g.:

HTML:

    <span class="mydate" data-date="1970-01-01">the beginning</span>

Config:

    ...
    item = {
    	created = "span.mydate",
    	createdattr = "data-date",
    ...

This will extract "1970-01-01" to the item.created, and not the element's data "the beginning".

feed
    object which defines how ratt will obtain the data for the feed details.

Example:

    feed = {
        title = "h1",
        description = "p.description",
        authorname = "span.author",
        authoremail = "",

feed.title
    feed title (can be css selector or lua function)

feed.titleattr
    attribute name of the selected element for the feed.title

feed.description
    feed description (can be css selector or lua function)

feed.descriptionattr
    attribute name of the selected element for the feed.description

feed.authorname
    feed author's name (can be css selector or lua function)

feed.authornameattr
    attribute name of the selected element for the feed.authorname

feed.authoremail
    feed author's email (can be css selector or lua function)

feed.authoremailattr
    attribute name of the selected element for the feed.authoremail

item
    object which define how ratt will obtain the data of the feed items. item data are obtained in parallel.

item.container
    individual items are selected with this attribute. This selector extracts the html elements that contain all the data that the item needs. Later item attributes run against this element. So if the items are all in a div e.g.:

    <div class="article-box">
    	<a href="https://article/url"><h3>example article title</h3></a>
    	<p>some article description</p>
    </div>

    The selector will be:

    item = {
    	container = ".article-box",

    Container can be also just a number or a lua function. In that case it is required, that the lua function will return a integer, with the number of items.

item.title
    item title (can be css selector or lua function)

item.titleattr
    attribute name of the selected element for the item.title

item.link
    item link (can be css selector or lua function)

item.linkattr
    attribute name of the selected element for the item.link

item.created
    item created date (can be css selector or lua function)

item.createdattr
    attribute name of the selected element for the item.created The date data which must be then parsed with the createdformat

item.createdformat
    the date format of item.created ratt uses the go time format https://pkg.go.dev/time#pkg-constants e.g.:

    item.createdformat = "01/02 03:04:05PM '06 -0700"
    item.createdformat = "Mon Jan 2 15:04:05 MST 2006"

item.description
    item description (can be css selector or lua function)

item.descriptionattr
    attribute name of the selected element for the item.description

item.content
    item content (can be css selector or lua function)

item.contentattr
    attribute name of the selected element for the item.content

item.image
    item imate url (can be css selector or lua function)

item.imageattr
    attribute name of the selected element for the item.image

nextpage
    if you want to extract data from more then one page, you can specify the selector for a link to the next page. The link will be requested and above selectors will be applied recursively.

nextpageattr
    attribute name of the selected element for the nextpage

nextpagecount
    an integer on how deep to follow the nextpage

sort
    is a enum for sorting the extracted feed items.

    Possible values are: REVERSE, CREATED_ASD, CREATED_DESC, TITLE_ASD, TITLE_DESC

  ERRORS

Returning errors is done with the error() function.

    if err ~= nil then
    	error("some error: " ... err)
    end

  LUA FUNCTION ARGUMENTS

Every lua function has two arguments by default:

sel
    the local goquery.Selection in feed attributes it is the entire html document in item attributes it is the item container element

index
    if the item.container is lua code and returns the number of items, the index variable is available in the item attributes and is the index the processed item.

    feed = {
    	title = function(sel, index)
    		local title = sel:find("h1"):text()
    		title:sub(index+2) --just a weird example
    	...
    item = {
    	...
    	title = function(sel, _)
    		local itemTitle = sel:find(".title"):text()
    		return mytitle .. ": " .. itemTitle

  LUA GLOBAL VARIABLES

url
    a url object, with the parsed value of the url with which ratt was called. It has methods: scheme, host, path, query, url

    item = {
    	container = function(sel, _)
    		--do some magic
    		return numberOfItems
    	end,
    	link = function(sel, index)
    		return "https://example.com/article" .. index
    	end,

  LUA LIBRARIES

Libraries are imported by default, no need for require(...).

goquery
    it's a subset of the great goquery library (https://github.com/PuerkitoBio/goquery).

goquery.newDoc(STRING)
    takes a string of the html document and parsed it to goquery.Document

goquery.newDocFromURL(URL)
    parses a document from a url. Returns goquery.Document.

document.find(SELECTOR)
    find gets the descendants of each element in the current set of matched elements, filtered by a selector. It returns a new Selection object containing these matched elements.

selection
    represents a collection of nodes matching some criteria. The initial Selection can be created by using Document.Find, and then manipulated using the jQuery-like chainable syntax and methods.

selection.find(SELECTOR)
    finds the descendants filtered by a selector

selection.each(function(index, selection))
    each iterates over a Selection object, executing a function for each matched element. It returns the current Selection object. The function f is called for each element in the selection with the index of the element in that selection starting at 0, and a Selection that contains only that element. example:

    sel:find(".something"):each(function(i, s)
    	value = s:text()
    end)

selection.attr
    attr gets the specified attribute's value for the first element in the Selection. To get the value for each element individually, use a looping construct such as Each or Map method. selection.first

selection.parent
    parent gets the parent of each element in the Selection. It returns a new Selection object containing the matched elements.

selection.text
    text gets the combined text contents of each element in the set of matched elements, including their descendants.

selection.html
    Html gets the HTML contents of the first element in the set of matched elements. It includes text and comment nodes.

gojq
    it's a subset of the gojq library (https://github.com/itchyny/gojq) which parses a query and extracts data from json.

gojq.parse(STRING)
    parse parses and returns a gojq.Query

query.runMap(MAP)
    run the query on a map object

query.runArray(ARRAY)
    run the query on a array object

    --extract title from json
    query, err = gojq.parse(".title")
    title, err = query:runMap(sel:text())
    if err ~= nil then
    	error("gojq runMap:" .. err)
    end

EXAMPLES

Full example of extracting feed from apnews.com:

    ratt.add(
    	"https://apnews.com/hub/*",
    	{
    	  feed = {
    		title = "head title",
    	  },
    	  item = {
    		container = ".FeedCard:not(.HubPeek)",
    		title = "h3.-cardHeading",
    		link = function(sel, _)
    		  local l = sel:find("a[data-key=card-headline]")
    		  return "https://apnews.com" .. l:attr("href")
    		end,
    		linkattr = "href",
    		created = function(sel, _)
    		  ts = sel:find("span.Timestamp"):attr("data-source")
    		  return ts
    		end,
    		createdformat = "2006-01-02T15:04:05Z",
    		description = "div.content p",
    		imageattr = "src"
    	  }
    	}
    )

Some explanation:

item.container
    css selector which gets all elements that have the class FeedCard but not the class HubPeek

item.title
    h3 elements with -cardHeading class

item.link
    lua code that gets all <a> elements with attr data-key=card-headline. the links on the website are local, so we must prepend the host "https://apnews.com"

item.created
    get the span element with the "Timestamp" class and then get it's "data-source" attribute

item.createdformat
    the format to parse the item.created value

SEE ALSO

ratt(1)

AUTHORS

Maintained by Vladimír Magyar <vladimir@mgyar.me>, who is assisted by other open source contributors. For more information about ratt development, see https://sr.ht/~ghost08/ratt and bugs/patches can be submitted by email to ~ghost08/ratt@todo.sr.ht