Sounds like the perfect application for Huginn . I wrote up a recipe for running under docker (swarm mode optional) here : https://geek-cookbook.funkypenguin.co.nz/recipies/huginn/
From a quick look at the site, most likely it is due to the site being built on a js framework (i.e. React) and needs to be rendered on the client first before it can be parsed.
There are wiki pages for Huginn about doing browser emulation, you should check them out.
https://github.com/huginn/huginn/wiki/Browser-Emulation-using-browserless
Yes, I've been using n8n for a while and I really like it.
I also use Huginn, which I like even more in some ways. Not only because of its name and logo, but that also is a factor. :D https://github.com/huginn/huginnI think Huginn can do everything n8n can.
Basically, I use n8n for things that look more like data transformations and the like (think small ETL workloads), because it has a better editor for them.
And I use Huginn for almost everything - it acts as a processor and orchestrator for most of my processes. (It also triggers my n8n workflows when needed, so I have all the controls in one place.)
Eventually, you will want to set up a server that runs a that does this kind of thing like scrape sites for proxies and test them. There are a few scripts to automate it (haven't tried this one yet.) you can always get that free AWS tier running Debian or RH. You can also add on a website watcher and recon task so your server doesn't have to be just scraping proxies. look for a script that can test the proxies too so you cut time in half.
I wanted to try it on rpi before but I eventually gave up. I have other things installed too. https://github.com/huginn/huginn/wiki/Running-Huginn-on-minimal-systems-with-low-RAM-&-CPU-e.g.-Raspberry-Pi
I'm not sure what you expect exactly, but take a look at Huginn
You should be able to connect the PeakDetectorAgent, the EmailAgent and some content providers like search engines (check the Duckduckgo API for example) or Twitter.
I am using Huginn to scrape the Flying Activity website once a day and email me when the latest posts match Canberra or Russell.
Unfortunately there's no mention of either in the details for this one, so I'll have to update it to match ADFA and Campbell
TL;DR - Doing my own thing, doesn't quite work yet.
I can't speak to all of your concerns, but for 2&3 you should consider looking at RSS Bridge and Huginn. Properly configured these tools can act as a kind of pocket knife for feeds.
https://github.com/huginn/huginn is one such tool. (It does this and much more, as it can trigger agents or notify you.)
Keep in mind that if the site doesn't publish something like an RSS feed, they've probably made a business decision that is better for them if you have to visit the site directly than to subscribe to a feed.
Have a look for sites which generate an RSS feed from your desired channels, but keep in mind you'd now rely on a third party able to see your subscriptions.
Unfortunately, many software projects these days target Docker as their only supported platform. Maybe you are an old wheel and think, I'll just install it "by hand" since I have the instructions in the docker file, right? Then you'll often discover that it requires old versions of dependencies that are long gone from ports, or supporting software that has never been ported.
Install Huginn by hand on FreeBSD and let me know how it goes for you.
You might counter that that sounds like shit software that probably has a lot of security issues hidden in the black-box image that no one is looking at, and a lot of lazy programing that will surely manifest as bigger problems down the road. You might say that slouching into a platform mono-culture is deeply and dangerously unhealthy, and ripe for shocks and plagues.
Probably all true, but now you are running linux too because the app you need today isn't installable on FreeBSD.
Usually you build task queue system to poll services for new data with settings configured by user or setup event subscription system to get data when it changes.
Study code in Huginn (https://github.com/huginn/huginn) or n8n (https://github.com/n8n-io/n8n). There are other systems too.
Maybe you should specify which features are desired.
DWKit (last commit 5 month ago)
[n8n (not really open source)](n8n.io)
This project could work for your situation:
https://github.com/huginn/huginn
I use it in a Docker container.
You will have to test yourself, but Huginn has Webhook support. So you could see if it can load data to NC with;
https://apps.nextcloud.com/apps/webhooks
You could do it with Huginn. Set up a bunch of RSS Agents to pull the ones you listed. Run each event through an event formatting agent to pick out the parts you care about. Run those events through a Data Output Agent configured to emit an RSS feed.
Thank you.
I am looking forward testing Airflow to see how it fits into this whole automation landscape.
Huginn and n8n seem to be quite interesting but I have not yet evaluated them so I cannot fully understand how they compare. Great to know another option for a test candidate now.
Did you by any chance look into alternatives before deciding to go with Airflow and if you did what was the base for this decision?
I've started playing with Huginn. It can monitor elements of a page (please someone tell me how to monitor any changes...) and send you a notification to email.
You could check out Huginn. It requires some learning curve, but once you get the hang of it, this can be your personal information hub. It can be used similarly as IFTTT, though it is not as user friendly and sadly jt doesn't support as much services. But I use it daily and for me it's the best thing after sliced bread.
https://github.com/huginn/huginn
can do lots of what IFTTT does but takes some effort to install and you might need to pay hosting fees if you've not got a server already.
IFTTT. I use Huginn and some related services, but they're all a nightmare for user-friendliness in comparison.
If you have some familiarity with coding, you can look up huginn (https://github.com/huginn/huginn) - it'll let you set up a custom RSS aggregator with whatever search filters you'd like, and output the results as your own RSS feed. I use it to take in RSS feeds from ~100 journals and output a summary feed to Feedly with papers matching my keyword searches.
I used huginn to create watchers for products I wanted to buy in the next few weeks/months. As soon as the product has a significant price drop (let's say 10%) I received an email.
Recently I found out, if you add a product to your wishlist, Digitec notifies you via email if the price drops.
Definitely huginn (similar to) or Trigger Happy.
I use Huginn to process all of my stuff. From there I have notifications to Discord for certain communities (some of which I'm not really even a part of...) and notifications to Telegram for personal stuff.
To anyone who comes across this thread. If you use Huginn and want to share any creative or useful things you come up with, please feel free to stop by /r/youragents and share. 👍🏻
I guess I just don't understand your argument.
Cool, you don't use docker. How does that make you immune to software vulnerabilities? You still have to granularly manage every service you run when it's on bare metal.
I also don't see how they "only care about docker". There's directions in the readme on how to install locally.
Yeah sure, keep saying Linux is becoming like Windows. Windows locks you into one OS, where as containerization let's you use whatever you want.
If devs have to spend 80% of their time making their niche software work on every conceivable platform with specific builds, how are they supposed to make actual features?
https://github.com/huginn/huginn
This, I was able to get it to run through windows toolbox, but I was just curious how can I update it through docker after downloading it off git to test my changes.
I run in a digital ocean vps (no docker).
Advice would be to name everything in a meaningful way, a lot of times you will have an idea and need 3 or 4 agents to complete it. So if they are all named in a way you can understand months later, your life will be better.
Also, the javascript agent solves a lot of limitations I've had with state, I sometime use it as key-value store for a very simple scripts.
Make the daily digest at least ever other day or less often.
I have a postmark account that I use email to json to huginn and its amazing. For example gmail forwards my comcast, and rent receipts to huginn and it checks that they were paid (gap dector) and puts in the digest.
Switch to what ever the alternative diagram system is. The default one can only show 50? agents and doesn't link to them when you click them. It was a few config changes for me. https://github.com/huginn/huginn/issues/1664
Use Credentials! They will make your life easier in the long run
Oh, great, I actually saw your post from about a year ago, but didn't get to reply back then! Cool job! Yep, I understand pain about not having rich fields. To be fair, it's a tradeoff: even if you are a coder, in certain circumstances using visual tables (e.g. airtable/spreadsheets) and workflows (e.g. zapier or huginn) is super convenient for understanding and messing with data. But on the other hand if the tool misses some basic features, you can't do anything about it.
I use Huginn to do website monitoring for me (and a slew of other things). Some site changes go to a digest, others go to slack alert, txt message, or push notification.
I've heard good things about huginn as a self hosted ifttt alternative: https://github.com/huginn/huginn Looks like it has an "IMAP folder agent" which might do what OP wants, I haven't used it myself so I can't say much more.
I'm hoping to work on a personal activity tracker that will help me monitor how I interact with the digital world. Something along the lines of https://github.com/huginn/huginn but with an improved interface.
I could recommend huggin but according to the developers themselves the software does not run properly on a raspberry pi.
I've not used it but have you seen https://github.com/huginn/huginn ?
Although the quote from https://www.makeuseof.com/tag/ifttt-alternatives/ says "I’ll warn you now: if you’re not a skilled programmer, steer clear. Huginn is hugely complicated for beginners. Creating and setting up agents is a difficult process."
There is nothing magical about IFTTT- its just a bunch of calls to web API's but the devil is in the detail of getting each service working in a consistent fashion. IFTTT has been able to leverage vendor cooperation to achieve that in a way that an open-source project probably couldn't.
Thanks for sharing your experience. I don't want it as a workstation, I want it most importantly as a home server (voice controlling hue lights and wifi plugs through VoiceAttack + nodejs is one of the projects I have in mind), then using it as HTPC. Virtualization would be a cool feature, I would like to setup something like huginn. I won't be using virtualization for a destkop experience, just as a server: boot up a linux machine, setup everything through ssh from another machine and then leave it running. My concerns are if with everything running at the same time will I have issues with the HTPC part? I am not planning to stream 4k content, but it should be able to flawlessly play 1080p video from youtube, Twitch and BitTorrent streaming. Thanks for your help.
URL: flashfed.com
Purpose of Startup: Aggregate flash sales and rank them with machine learning. Help people save money!
Technologies Used: Azure ML, Huginn, Wordpress.
Feedback or Support Requested: Feedback and public beta users.
Comments: What do you think of the UX? Any comments are appreciated.
No problem bro, almost everything is automated using huginn. I scrape the sites, format/modify the data, send a post request to an Azure ML web service to score the sale, which is all finally output as rss. My wordpress install then imports the data as posts. You can check it out at https://flashfed.com
Now I just have to convince people to use the damn thing.
URL: flashfed.com
Purpose of Startup: Aggregate flash sales and rank them with machine learning. Help people save money!
Technologies Used: Azure ML, Huginn, Wordpress.
Feedback or Support Requested: Feedback and public beta users.
Comments: What do you think of the UX? Any comments are appreciated.