Thoughts on Integrating Systems & IoT

Posts tagged “rss

Creating a Web Service in Azure, Part 1- Introduction and Architecture


In a series of articles I will describe how Azure can be used to power a back end for a front end application.

My front end application is a simple Rss aggregator.  I follow a number of blogs and I currently use a Windows forms application that I wrote to monitor for new content: the application periodically downloads Rss feeds and processes them on the client side so I can then read them offline (the feed content is stored in a SQLite database).  I would like a better solution that can run on my Android phone and also where feeds can be synced even when the client application isn’t running; the back end will instead be responsible for downloading latest feed content and aggregating the data.

A key design feature is that the new Android client will be lightweight: it will carry out minimal processing of the data and won’t be responsible for downloading feeds from the various blogs.  Such a setup was passable for my high spec laptop but won’t do for my much lower spec phone for these reasons:

  • Downloading feeds from the various blogs will blow out my data plan.
  • Heavy processing on the client side will consume precious CPU cycles and battery power, making my phone slow/unresponsive with a constant need to find the next power outlet to charge it.

So with these limitations in mind, the back end will instead do the “heavy lifting” of downloading and processing feeds and ensure that sync data is optimized to the needs of the client, so minimizing bandwidth consumption

I must also mention as well that while thinking on how Azure could be used to power a back end service, a two part article was published in MSDN magazine that is pretty much along the lines that I was thinking for my own web service (please see the “References” section below for links to these two articles).  The MSDN articles describe a service that aggregates Twitter and StackOverflow data intelligently, while my proof of concept aggregates Rss feed data from blogs, for example.  I draw on these 2 articles heavily in the series.

Another major advantage (mentioned in the MSDN article series) of a cloud back end is better scalability: instead of each client downloading and processing the same feeds individually, the back end application can do this in a single operation, getting around any throttling limitations that may be imposed on some web services.  So as the popularity of an app increases, this doesn’t result in a related decrease in performance (due to throttling) which would damage the reputation of the app.


The diagram below shows a high level overview of the solution:

Datamate Architecture

Figure 1  Datamate Architecture (Based on Figure 2 in Reference Article [1])

Some of the key features of the architecture are as follows (walking through the diagram from left to right):

  • Azure SQL Database is used to store Feed data in a relational database and the data is accessed using Entity Framework (EF) via an internal data provider API.  It is envisaged that as further data sources come on board (other than just Rss feeds) each data source (e.g. Twitter) will have it’s own provider API that is implemented to the requirements of the particular data source that is onboarded.
  • Azure WebJobs represent the worker processes – they run as a scheduled background task, downloading and processing Rss feeds and writing the results to the database.
  • A REST API, implemented using ASP.NET Web API, provides an interface for clients to retrieve data.
  • A simple client app (mobile and web) will use the REST API to download data and maintain a client side cache of the data, to the preferences specified by the user, once authenticated and authorised by the REST API.

That’s it for now – stay tuned for part 2!!  In the next post, I will discuss the design and development of the Azure SQL Database and Azure WebJob that represent the “backbone” of the solution.

As always, any comments or tips most welcome.


[1] MSDN Magazine Aug 2015 Issue, Create a Web Service with Azure Web Apps and WebJobs, Microsoft.  Available from:

[2] MSDN Magazine Sep 2015 Issue, Build a Xamarin App with Authentication and Offline Support, Microsoft.  Available from:


Introducting jarwin – “Just Another (Rss) Reader for Windows”, Beta

This is a post to introduce a personal project I have been working on for the last few months and announce the availability of an early beta version, which I hope people will download and provide feedback to me on.

I have learnt an enormous amount over the last few months working on it and had lots of fun along the way (and some frustrations too!).

If you would like the short version of this post, the app is available for download here.

Also check out the wiki here.

Unfortunately a known issue/limitation is currently no ability to specify proxy server credentials, so you will be probably out of luck trying to sync on a corporate network.  This will be a new feature in the next release…

Also issues/bugs can be logged here.  I’m a very busy man these days so I will try and reply as promptly as I can.

What is this app?  Well it’s not much really – it’s a (very) simple Rss reader and a Windows forms application.  I use it to follow blogs that I’m interested in.

As I mention in the app wiki, it represents only a small percentage of the vision I have in mind, which is an app that I can use to download content from various disparate data source (email, blogs, Twitter, Facebook, LinkedIn etc) and present all this information in an intelligent and meaningful way, such that I can view at a glance and get answers to questions such as: what are the key trends in the systems integration space right now?  What information is meaningful to me?  What is the impact of blog post x?

It would also be possible to “swap” datasets in and out of the analytics (quickly and efficiently).

These ideas are nothing new, I know…  There are tools out there already too.  But I’m fed up using other peoples software outside of my work domain/practice – I would much rather write and use my own.

The seed for these ideas come from a common problem pervasive in all industries I believe: masses amount of data, that is constantly evolving and changing.  How is one supposed to digest and make sense of all this data?  I follow a number of blogs and Twitter feeds, for example.  But I myself feel overwhelmed with the amount of “noise” out there; recently I have stopped following my Twitter feed, since although there is a lot of interesting and useful information in it, there is also a lot of stuff I’m not interested in – life’s too short to trawl through that, let alone try to link it with other data sources.  (I remember reading somewhere that Donald Knuth decided to no longer bother with email, for this reason).

So one day, over an early morning coffee, I hope to be able to view quickly and intelligently, a consolidated view of all my data feeds.  To have a view also, that is not just a mass of text but a complete model that (most importantly) makes sense to me.  Anyway, there is quite a way to go to reach this goal!!