Here's how you would use it

go get github.com/jonathan-kosgei/geodist
gopy build -output=geodist -vm=python3 github.com/jonathan-kosgei/geodist

then you can import the module in Python

>> from geodist import geodist
>> dir(geodist)
['Coord', 'VincentyDistance', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_collections_abc', '_geodist', 'collections', 'currentdir', 'cwd', 'go', 'inspect', 'os', 'sys']

Naturally I was curious whether Python extensions written in Golang have comparable performance to those written in C, which is what we explore in this post.
I'll be testing the Vincenty algorithm for calculating the distance between 2 co-ordinates using:

1. Native Python - using the geopy module

2. Golang extension created with GoPy - I forked John Taylor's geodist package and slightly modified it to make it compatible with GoPy

3. C extension - using uvincenty

Here are the results using iPython

>> from geodist import geodist
>> import uvincenty
>> import geopy.distance

>> boston_go = geodist.Coord(42.3541165, -71.0693514)
>> newyork_go = geodist.Coord(40.7791472, -73.9680804)

>> %timeit geodist.VincentyDistance(boston_go, newyork_go)
6.24 µs ± 29.2 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

>> boston = (42.3541165, -71.0693514)
>> newyork = (40.7791472, -73.9680804)

>> %timeit uvincenty.vincenty(*boston, *newyork)
732 ns ± 2.98 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

>> %timeit geopy.distance.geodesic(boston, newyork)
190 µs ± 1.51 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

The CPython extension (uvincenty) is ~260 times faster than the pure Python implementation from geopy.
The Golang Python extension (geodist) is ~30 times faster than the pure Python implementation.
The CPython extension is 8 times faster than the Golang Python extension.

I'm not certain why the Golang Python extension is slower than the C version given that GoPy generates a CPython extension, my guess is that the generated code is less efficient than the native C approach. However this still shows that a sizeable speedup can be achieved using GoPy generated Python extensions with the added bonus of lowered development time and fewer memory bugs by coding in Golang.

Thankfully, online businesses can establish and hold onto that personal connection by using IP geolocation. Not only can local businesses provide products and services to a wider customer-base, but they can also identify their local customers and develop/maintain a relationship with them.

Naturally, there are many companies that provide IP geolocation services, and choosing one can be a balancing act between price, reliability, and speed. But fear not; we’ve composed the following list of the 10 best IP geolocation services for you to review.

1. IPData.co
2. Maxmind
3. Abstract’s IP Geolocation API
4. IP Geolocation API
5. ipapi.co
6. ipstack.com
7. ip-api.com
8. ipgeolocation.io
9. IP2Location
10. ipify.org

IPData.co

When it comes to geolocation data, IPData’s API provides just about everything you need without breaking the bank. In addition to accurate, basic geolocation data such as continent, city, region, etc, IPData provides threat intelligence data, ISP and company information, and internationalization data like currencies, time zones, and languages. While other services would charge extra for this additional information, IPData exposes all of their data across all pricing tiers. This sounds like a lot of data (it is), but you can easily filter your request results to just the information you need.

IPData’s services are trusted by many large corporations, and they are hosted in 11 AWS data centers around the world. So, it is reliable and fast— averaging 60ms per request without caching or CDNs.

IPData is very developer-friendly. The API is well documented, offering a complete list of URLs and query parameters, as well as useful examples to get you started. You can also consume their API by making your own raw HTTP requests, or you can use one of their official libraries for C#, Python, JavaScript, Go, PHP, and Java.

Pricing starts at $10/month for 2,500 daily requests, but they also offer a free tier for testing and non-commercial use. Unlike many other services, you can lookup multiple IP addresses in a single request, making efficient use of your daily limit.

Maxmind

Maxmind is regarded as one of the best IP address lookups and geolocation APIs. They offer an array of products, ranging from IP geolocation web services to downloadable databases that you can use in your applications.

Their GeoIP2 web services are served from four endpoints (hosted behind Cloudflare) in the USA, UK, and Singapore. Maxmind uses latency-based routing that supposedly sends your requests to the region closest to you.

Maxmind’s pricing structure depends upon the product you choose. Their web services use a metered model, where you pay for each request you send to the service, and different types of requests have different costs. Their downloadable databases use a subscription model, or you can purchase a database outright without future updates. They do, however, have a free service and database dubbed GeoLite2, but you do have to create an account to use them.

IPinfo.io

ipinfo.io isn’t just an IP geolocation API company; they provide several APIs that they bundle into different monthly plans. Their geolocation API provides accurate basic geolocation data, and it is available in all plans—including including the free plan. The Basic plan includes more detailed ASN information, and higher-tier plans include even more data, such as company info, domains that are hosted at the IP, and much more.
ipinfo.io is hosted on Google Cloud, and their infrastructure automatically scales with customer demand. They also provide SSL encryption for all their data transfers, including the free plan.
Developers will find ipinfo.io’s APIs are simple to use and well documented. You’ll find information on authentication, filtering response, response payloads, and everything else you need to interact with the API. Better still, ipinfo.io maintains many official client libraries in a wide array of languages and platforms and provides sample code for each of them.
Pricing is separated into five plans:

• Free — has a rate limit of 40,000 requests per month and provides basic geolocation data
• Basic — $49 when billed monthly and allows 250,000 monthly lookups with expanded ASN data
• Business — $499/month for 1,000,000 lookups, and it allows access to more data sets and APIs
• Premium — $1499/month for 2,500,000 lookups and even more data sets/APIs.
• Enterprise — Flexible amount of lookups and all data sets/APIs. You’ll need to contact them for pricing.

IP-API

Yes, you did read about ipapi a few moments ago. No, it is not the same as this provider. IP-API provides a simple-to-use interface for retrieving basic geolocation information about an IP address. Unlike many on this list, IP-API enforces a rate limit of 45 requests per minute. They do, however, provide a bulk endpoint to process multiple IP addresses in a single request.

It is hosted with Total Uptime Technologies, LLC, and while it is unknown how many servers the service uses to serve requests, IP-API’s status page provides information such as uptime (near 100%), latency (15ms average), and requests per minute (21,322 average).

For developers, IP-API provides thorough documentation regarding the endpoints, their expected and optional parameters, and the response. It supports multiple response formats, such as JSON, XML, CSV, and newline.

IP-API is a simple API with three simple pricing tiers. The free tier is available for non-commercial use, but it lacks SSL support. The Pro tier is €13.3 per month and offers SSL support and other features. The Enterprise tier comes with custom terms and agreements, but you have to contact the sales department for further information.

Abstract IP Geolocation API

Abstract is less of an IP geolocation company and more of a general API company, offering APIs for looking up IP address, exchange rates, email validation, and so much more. But don’t let that dissuade you; like many other entries in this list, Abstract’s geolocation API is used by many large corporations.

Abstract’s IP Geolocation API is hosted in AWS and built for scale and speed. However, Abstract does not let you filter your request results to save bandwidth. Therefore, every response contains the full JSON payload whether you want it or not.

The IP Geolocation API is one of the simplest on this list. Simply provide your API key and an optional IP address in order to perform a lookup. As a result, the documentation is very sparse. They provide detail regarding the response payload, but they only offer rudimentary code samples. There are no official libraries for any language or platform, but one could argue that such a simple API does not need them.

Pricing begins at $9/month for 200,000 requests and 50 requests per second. Abstract also offers a free tier of 20,000 requests per month and 1 request per second. Their other APIs are not included and have differing pricing structures.

IPGeolocationAPI.com

Not to be confused with Abstract’s product of the same name, IP Geolocation API is a free real-time IP geolocation service. It requires no sign-up or API key. The service is open source, and you could easily host your own service using the freely available code.

IP Geolocation API is hosted behind Cloudflare to ensure speed and reliability, and the geolocation data returned by the API is provided by a monthly updated MaxMind Geolite2 database.

The documentation is very limited—providing only a sample response payload and a few example URLs. There are no code samples or libraries.

ipapi.co

ipapi is yet another IP geolocation focused API company providing fast and accurate results. Its service is hosted in AWS behind Cloudflare; so, speed and reliability should not be a factor. In testing, the service responded with satisfactory latency (74ms).

For developers, it provides a thorough documentation, detailing the various end points and response payloads. There are code samples for many languages (PHP, C#, JavaScript, Ruby, Python, and others), and they provide official libraries for Python and Node.js/NPM. Unlike other services, ipapi supports payloads in JSON, JSONP, XML, CSV, and YAML.

ipapi uses a monthly pricing structure starting at $12 for 60,000 requests per month with no daily limits. They do provide a no sign-up required free tier that is limited to 30,000 requests per month with a 1,000 request daily limit. Their free tier, however, is suitable only for testing and development; it is not recommended for production.

ipstack

Like many other entries in this list, ipstack’s IP geolocation service is used by thousands of developers and trusted by large and small companies around the world. They provide a wide range of geolocation information; however, it is separated into different modules that are only available within certain pricing tiers.

ipstack provides three endpoints for standard, bulk, and requester lookups, all of which are hosted behind Cloudflare. The documentation for these endpoints are well done, detailing the required pieces of information (such as the API key), as well as various options for specifying the output format (JSON or XML) and response fields. Unfortunately, the sample code is sorely lacking, and there are no official libraries.

Pricing starts at $9.99/month for the location, currency, time zone, and connection modules. Higher priced tiers are available that include the bulk endpoint and security module. Their paid tiers are protected via SSL; so, you don’t have to worry about the request and response data being transmitted in the clear. The free tier, however, does not use SSL. Therefore, your data is at risk at being intercepted and read.

ipgeolocation.io

The ipgeolocation.io API provides basic geolocation information about an IP address, but it also exposes three complementary APIs for retrieving an IP’s time zone information, its astronomical data, and a user agent API; the latter, of course, is only useful when called from the browser. Despite having different end points for each of these APIs, they are not charged separately; they are included with every pricing tier.

ipgeolocation.io is hosted behind Cloudflare; so, you can expect it to have good reliability and response times. According to their API’s status page, it has near 100% uptime and a global response time average of 109ms.

Each endpoint is well documented, providing all of the information you need to make raw requests. They also provide code samples and official libraries for Java, C#, PHP, TypeScript, JavaScript, and jQuery.

There are several pricing tiers. They provide a free tier for developers with a 1,000 daily request limit and 30,000 requests per month. Their Bronze tier is $15 per month for 150,000 monthly requests. Their highest tier, Platinum, is $500 per month with 20,000,000 requests per month. They have so many pricing tiers that you are bound to find one that fits your needs.

IP2Location

IP2Location is one of the oldest IP geolocation services on the internet. They provide 24 IP geolocation databases and provide a number of proxy detection databases as well. These are separate products and not bundled with the API. IP2Location claims to use a “non-intrusive IP location lookup technology that retrieves geolocation information with no explicit permission required from users”, which of course is marketing-speak for typical IP geolocation look-ups.

IP2Location’s geolocation API is hosted behind Cloudflare, and I experienced an average of 72ms response time.

The API itself is more complex than it needs to be. IP2Location has 24 predefined payloads that you can specify in your request, and you’ll have to consult the documentation to determine which payload you need. There are also add-ons that provide additional information, but not all add-ons can be used with all packages. The documentation does spell it all out, however, and there are basic examples for PHP, Java, C#, Python, and Ruby. JSON and XML are supported payload formats.

Like the API, pricing is more complicated than it needs to be. Instead of monthly or yearly subscriptions, you purchase credits that are good for one year. Each request consumes credits, and the amount of credits is determined by the package and add-on for that request. The cheapest package is $49 for 100,000 credits.

ipify

ipify’s geolocation API is relatively new, but they claim to have more than 15 million IP blocks and their associated locations in its database. According to their FAQ page, their database has the highest number of unique locations from the USA, France, and Germany. In addition to their geolocation API, ipify also has a downloadable geolocation database and an IP proxy data feed for detecting anonymous users, but they are separate products and not part of the geolocation API.

Like many on this list, ipify’s geolocation API is hosted behind Cloudflare. In my tests, ipify’s geolocation API had an response time average of 64ms.

ipify’s geolocation is fairly simple, providing basic geolocation information; so, its documentation is thorough and concise. There are simple code samples for JavaScript, Java, C#, Perl, PHP, PowerShell, Python, and Ruby.

Pricing is broken down by the number of requests per month and the length of subscription. For example, 10,000 requests per month is $10 with a monthly subscription and $79 for a yearly subscription. More expensive tiers do not provide extra features—only more requests.

Conclusion

IP geolocation APIs are a must-have for any business that wants to know the specific locations of their customers. This information can be vital, especially for small and local businesses that want to maintain the home-town feel. They can also be useful for displaying relevant advertisements, as well as filtering risky or questionable users.

No matter what your goal is, the APIs in this list will serve your geolocation needs.

Autonomous Systems (AS) are used in BGP to route traffic to the correct network. ASNs are the notation for identifying a given AS and ASN data is stored by Regional Internet Registries (RIRs), which are used to look up where to route traffic. Outside of BGP routing, they may sound useless, but they’re actually great sources of data and they can be used to identify the organisation which controls a set of IP addresses. There are currently over 92,000 registered ASNs.

Which organisations use ASNs?

Internet Service Providers

The most well-known users of ASNs are Internet Service Providers (ISPs), such as Comcast, Virgin Media, Vodafone or Telefónica. They are allocated blocks of IP addresses by a RIR, which they then allocate to individual users. Your home internet’s IP address will be a part of a larger block of IPs, for example many Virgin Media users will have an IP within the ASN AS5089, which has a route, or CIDR block, of 81.106.0.0/15 , meaning the IPs range from 81.106.0.0 to 81.107.255.255.

Hosting Providers

Web Hosting Providers, such as AWS, Azure, and Oracle Cloud, need a lot of IP addresses so they can allocate them to their customer’s servers. When you launch a server with a public IP, just like your home internet, it’ll be given an IP address from a larger pool. AWS is such a large provider that, based on their public list of IP ranges, they’re allocated 109,845,950 IPv4 addresses, which is approximately 3% of the global total.

Anyone else!

The list of organisations who use ASNs could go on for ages. Anyone who needs a large block of IP addresses could use them. Other notable organisations include Netflix, Apple, Microsoft, but they’re also used by some smaller enterprises to keep all their office networks within the same IP range, which can help with security.

Why are ASNs useful?

ASNs are essential to the operation of the Internet, but most people don’t need to understand the inner-workings of them. However, ASNs can still be incredibly useful as they associate a block of IP addresses with an organisation, which means we can look up organisation information by IP address.

Who is accessing my website?

One of the most common uses of ASN data is to find out where your visitors are coming from. Along with geolocation data, many popular analytics tools will show which ISPs/organisations are accessing your website. All this information is powered by ASN data.

Could this IP address be malicious?

We can use ASN data to help us identify IP addresses as malicious. For example, if there’s an abnormally high number of login attempts, we can look up the IP addresses which are making those requests. If the IPs are from ASNs associated with hosting providers, they’re likely trying to gain access to your users’ accounts! Additional information, such as ipdata’s threat API and location data, can help assist this.

Who is serving this website?

Using a DNS lookup, we can find the IP addresses which are serving a given website. This can help us to find the hosting provider, and might indicate if the site is malicious. The IP addresses can be found using an online tool, like dnslookup.org or the dig Unix command, for example dig +short ipdata.co.

How can I look up ASNs?

There are many different ways to look up ASN data by IP address. We’ll explore 5 different options, each with different pros/cons. To give a consistent example, we’ll be testing an IP address associated with serving apple.com - 17.253.144.10.

Whois

Whois is a utility tool which can look up the data associated with a domain or IP address. It can be used on the command line in Unix or Windows-based systems.

whois 17.253.144.10

NetRange:       17.0.0.0 - 17.255.255.255
CIDR:           17.0.0.0/8
NetName:        APPLE-WWNET
NetHandle:      NET-17-0-0-0-1
Parent:          ()
NetType:        Direct Assignment
OriginAS:
Organization:   Apple Inc. (APPLEC-1-Z)
RegDate:        1990-04-16
Updated:        2017-07-08
Ref:            https://rdap.arin.net/registry/ip/17.0.0.0

OrgName:        Apple Inc.
OrgId:          APPLEC-1-Z
// ... More information - try it yourself!

By default, whois will query IANA to find which RIR to query for detailed information. In the case of Apple, it uses [arin.net](https://www.arin.net). Based on the response, we can immediately see that Apple owns the IP, and has done since 1990.

The ASN associated with the IP should be after OriginAS, but strangely for this IP, there’s no result! This could be an issue caused by the fact that Apple is assigned an entire /8 address block, the largest possible size, which means that any IP which begins with 17. will be from Apple! Regardless of the reason, we’ll need to try another tool to find the ASN.

Cymru

Cymru provides an easy-to-use whois compatible API. This means we can switch the whois host to whois.cymru.com and use it to look up IP addresses.

whois -h whois.cymru.com 17.253.144.10

AS      | IP               | AS Name
714     | 17.253.144.10    | APPLE-ENGINEERING, US

The output is short and easy to read - this IP address’ ASN is AS714, which is owned by Apple. However, it’s very limited and missing a lot of useful information, so let’s look at PeeringDB and CAIDA, which can enrich the data.

PeeringDB

PeeringDB is the go-to solution for ASN-owners to decide which ASNs to peer with. When an ASN is peered with another, it allows both networks to exchange traffic between each other. In Apple’s case, they encourage ISPs to peer with them to deliver Apple content directly to customers. For our use-case, this means they have a very detailed database of the organisations which operate each ASN, along with their relationships with each other.

PeeringDB has no functionality to look up a ASN data by IP address, but as we already know the ASN for 17.253.144.10 is 714, we can look up all the peering information about this IP - https://www.peeringdb.com/asn/714. PeeringDB also provides an API, so we can access this data programmatically, but frustratingly there is no dedicated ASN API. First, we’ll need the Network ID, which can only be found by searching for the ASN…

curl --write-out "%{redirect_url}\n" "https://www.peeringdb.com/search?q=AS714"

https://www.peeringdb.com/net/3554

We can then add /api to the start of the URL path to get the JSON API endpoint:

curl https://www.peeringdb.com/api/net/3554

{"data": [{"id": 3554, "org_id": 8418, "org": {"id": 8418, "name": "Apple Inc",
// ...

CAIDA

CAIDA provides a few tools and services associated with IP addresses, BGP and ASN data. AS Rank is the most relevant to us – it’s tool which allows searching by ASN and ranks them by the number of IPs that can be reached by following customer links.

Like PeeringDB, we can’t search by IP address, but we can search by ASN. The results for [AS714](https://asrank.caida.org/asns?asn=714&type=search)  show it, at the time of writing, to be ranked 6,630th. The top ranked ASN is 3356, owned by Level 3, which is connected to about 75% of all IPs.

AS Rank’s data is available via an API which supports GraphQL and REST, allowing us to flexibly query their systems. The RESTful API returns all their data associated with an ASN with a GET request.
curl https://api.asrank.caida.org/v2/restful/asns/714

{
  "data": {
    "asn": {
      "rank": 6630,
      "asn": "714",
      "asnName": "APPLE-ENGINEERING",
      "source": "ARIN",
      "cliqueMember": true,
      "seen": true,
      "ixp": null,
      "longitude": -118.278526797574,
      "latitude": 40.9090750557379,
      "organization": {
        "orgId": "e7117ce3ab"
      },
      "cone": {
        "numberAsns": 2,
        "numberPrefixes": 1022,
        "numberAddresses": 16816640
      },
      "country": {
        "iso": "US"
      },
      "asnDegree": {
        "total": 316,
        "customer": 1,
        "peer": 292,
        "provider": 23
      }
    }
  }
}

ipdata

The ipdata ASN API is easy-to-use and works with a simple HTTP request. It can look up a huge amount of information associated with an IP address, and it even enriches the ASN data with a usage type, helping to determine whether the ASN is allocated to an organisation, ISP, hosting provider, or something else!

For our Apple IP address, a request would look like this:
curl "https://api.ipdata.co/17.253.144.10/asn?api-key=test"

{
  asn: "AS714",
  name: "Apple Inc.",
  domain: "apple.com",
  route: "17.253.144.0/21",
  type: "hosting"
}

At the same time as providing ASN data, ipdata’s API can look up location information, detect VPNs, and more, providing you with an affordable all-in-one solution.

We’ve also written a handy guide – How to look up Company Details by IP address, which follows a practical example of using ASN data to enrich a “Contact Us” form with company information.

Conclusions

ASN data is critical for the Internet to function, but most people have no idea about BGP routing, AS’, or RIRs. Despite their obscurity, ASN data is valuable – it can identify the organisation associated with an IP address, showing us where website traffic is coming from and helping to identify malicious traffic. There are many tools and providers to look up ASN data, each with their own pros and cons, so we’d recommend trying out the options and choosing for yourself – we’re confident you’ll love ipdata!

Bots come in all shapes and sizes. Many are legitimate, such as Search Engine bots (or spiders), which crawl your website to include pages on search engines like Google. There are plenty of illegitimate bots too – one common issue is referral spam, advertising sites by making them show up in your web analytics, consuming your resources and usage limits.

Detecting bots

Legitimate bots

Legitimate bots will send a User-Agent which indicates that they’re a bot. Most of these include bot in them, such as Googlebot, but there’s a long list of rules and exceptions. One frustrating example is that Cubot is often detected as a bot, but it’s not a bot at all – it’s a mobile phone manufacturer.

For those reasons, it’s best to use a library to detect whether it’s a bot. Most major programming languages will have a library available – for Node.js, there’s a library named [isbot](https://github.com/omrilotan/isbot) and for PHP there’s one named [device-detector](https://github.com/matomo-org/device-detector).

Usage of isbot is simple:

const isbot = require('isbot')

isbot('Googlebot/2.1 (+http://www.google.com/bot.html)') // true
isbot('Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36') // false

Illegitimate bots

Referral spam bots, and other illegitimate bots, will try to disguise themselves as normal visitors to your website. For that reason, they won’t set a recognisable User-Agent, and it’s impossible to reliably detect all of them.

Luckily, there’s an open-source list of known referral spammers. We can compare the Referer header against this list and decide how to handle it…

const fs = require('fs')
// referral-spammers.txt downloaded from https://github.com/matomo-org/referrer-spam-list/blob/master/spammers.txt
const spammerList = fs.readFileSync('./referral-spammers.txt')
  .toString()
  .split('\n') // each spammer is on a new line
  .filter(Boolean) // filter out empty lines

const isSpammer = referer => {
  return referer && spammerList.some(spammer => referer.includes(spammer))
}

isSpammer('https://casino-top3.ru') // true
isSpammer('https://ipdata.co') // false

Bots which aren’t referral spammers will be very difficult to detect. It’s best to rely on threat data to ensure you’re able to block requests from IPs which have been flagged as malicious. ipdata’s threat API can help to do exactly this. Additionally, ipdata’s ASN API can detect IP addresses which are associated with hosting providers, such as AWS. Hosting IPs are often used for bots and hacking attempts, but they could also be legitimate proxies. Here’s a small script, which we’ll use later, to detect hosting providers and threats.

const axios = require("axios");
// Get an ipdata API Key from here: https://ipdata.co/sign-up.html
const IPDATA_API_KEY = "test";
const getIpData = async (ip) => {
  const response = await axios.get(
    `https://api.ipdata.co/${ip}?api-key=${IPDATA_API_KEY}`
  );
  return response.data;
};

// Let's test it out!
const microsoftIpData = await getIpData("13.107.6.152");
microsoftIpData.threat.is_threat; // false - this IP is not associated with threats
microsoftIpData.asn.type; // "hosting" - this IP is a Microsoft hosting IP

Responding to bot traffic

Now we’ve detected a majority of our bot traffic, we need to decide what to do with it!

Excluding bots from analytics

Most people want to exclude all bot traffic from their web analytics tools. Many tools have a built-in methods for excluding bots, but if you wish to control this yourself, you can exclude the tracking codes. This can help prevent bots from consuming your usage limits, and ensures consistency across different tools.

In Node.js with Express and EJS, we can do this with just a few lines of code.

**index.js**

const isbot = require('isbot')
const isSpammer = require('./is-spammer') // isSpammer function defined above
const getIpData = require('./get-ip-data') // getIpData function defined above
const express = require("express")
const app = express()

app.set('view engine', 'ejs')

app.get("/", async (req, res) => {
  const ipdata = await getIpData(req.connection.remoteAddress)
  if (ipdata.threat.is_threat) {
    // Block threats altogether!
    res.status(403).send('You are not allowed to access this site')
    return
  }
  const includeAnalytics = 
    !isbot(req.get('user-agent')) && 
    !isSpammer(req.get('referer')) &&
    ipdata.asn.type !== 'hosting'

  res.render("./index.ejs", { includeAnalytics })
})

app.listen(8000)

**views/index.ejs**

<html>
  <head>
    <% if (includeAnalytics) { %>
      <script>
        // Google Analytics here
        // Other analytics scripts too..
      </script>
    <% } %>
  </head>
  <body>
    Hello!
  </body>
</html>

Now, we can test our implementation using curl.

Browser with no referrer

curl -H "user-agent: A-Browser" http://localhost:8000
<html>
  <head>
    <script>
      // Google Analytics here
      // Other analytics scripts too..
    </script>
  </head>
  <body>
    Hello!
  </body>
</html>

Bot user-agent

curl -H "user-agent: bot" http://localhost:8000
<html>
  <head>
  </head>
  <body>
    Hello!
  </body>
</html>

Referral spammer

curl -H "user-agent: A-Browser" -H "referer: https://casino-top3.ru" http://localhost:8000
<html>
  <head>
  </head>
  <body>
    Hello!
  </body>
</html>

As you can see, when we request the website from a normal browser, they’re served the analytics scripts, but bots and referral spammers are excluded.

This can easily be adapted to totally block traffic from bots or referral spammers from accessing your website, you can treat them like threats (see line 11 of index.js), but this should usually be unnecessary for most sites.

Serving a Search Engine Optimised version of your site

Serving a different version of your website to crawlers can cause a major improvement in SEO. Modern websites, especially single-page-applications can be hard for crawlers to read and index, so we can serve them a simplified version.

The approach is very similar to excluding bots from analytics.

**index.js**

const isbot = require('isbot')
const express = require("express");
const app = express();

app.set('view engine', 'ejs')

app.get("/", (req, res) => {
  if (isbot(req.get('user-agent'))) {
    res.render('./seo.ejs')
    return
  }
  res.render("./index.ejs")
});

app.listen(8000);

**views/seo.ejs**

<html>
  <body>
    Hello search engine crawlers + bots!
  </body>
</html>

**views/index.ejs**

<html>
  <body>
    Hello normal visitors
  </body>
</html>

Again, we can test our implementation with curl.

Browser

curl -H "user-agent: A-Browser" http://localhost:8000
<html>
  <body>
    Hello normal visitors
  </body>
</html>

Bot

curl -H "user-agent: bot" http://localhost:8000
<html>
  <body>
    Hello search engine crawlers + bots!
  </body>
</html>

To ensure high performance, which can also improve your search rankings, you may want to use an Edge Computing solution, such as Lambda@Edge or Cloudflare Workers. For detailed guides on edge computing, check out our posts: Using Cloudflare Workers to redirect users based on country ****and Redirect users based on country with Lambda@Edge.

Conclusions

Detecting and adapting your website for bot traffic can be incredibly valuable. Simply excluding them from your analytics can reduce your usage costs and keep your data clean – bots can cause your conversion rates to drop, distracting you from focussing on real customers. Serving a specific site dedicated to SEO won’t be required often, as search crawlers are able to index most websites, but for highly-complex single page applications, it can have a huge impact on your search rankings.

Let us know your bot-detection use-cases in the comments below!

Today, it’s easier than ever for local businesses to not only reach new customers through the Internet, but to also cater to their local customer-base by offering promotions. With tools like ipdata’s location-oriented API, local shops can identify their hometown customers and offer customized incentives.

In this article, you’ll learn how to use ipdata’s API to show promotions for a visitor based upon their location. You’ll need an API key, and if you don’t already have one, sign up for a free ipdata account.

You’ll build a service class for a Laravel application. So, if you want to follow along and actually run this code, you’ll need to setup your environment with the following:

• PHP 7.3+
• MySQL (or comparable database, such as MariaDB)
• Laravel 6+ (it’s easy to install; just follow the directions)

Note that even though this article uses PHP and the Laravel framework, the same ideas and concepts can be applied to any language and framework.

Setting up the Project

The first thing you need to do is create a new Laravel project. The easiest way to do so is by running the following command in your terminal:

laravel new <project_name>

Replace <project_name> with your name of choice. Laravel, like many other popular frameworks, provides many command-line tools for managing different aspects of the project. So, leave your terminal window open.

Next, change the directory in your terminal to the newly created project folder, like this:

cd <project_name>

The next step is to install the official ipdata PHP library and supporting packages. Note that this is not a requirement—you can consume the ipdata API with your own code. However, the library saves a lot of time and makes it very easy to interact with the API. In your terminal window, execute the following command:

composer require ipdata/api-client

This command uses the composer package manager to download and install ipdata’s PHP client library to your project. The API client is built upon PSR standards, and you need to install some supporting packages with the following command:

composer require nyholm/psr7 symfony/http-client

Next, you need to configure the project to find your database. Open your project’s .env file and modify the following options:

DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=my_awesome_store
DB_USERNAME=root
DB_PASSWORD=

Naturally, these settings depend upon your database’s settings. Be sure to input the correct information; otherwise, the database migrations (as well as the application itself) will fail.

Creating the Promotions Table

Your project will track the various promotions it provides visitors with a single database table called promotions. While you can manually create your promotions table with any number of database tools, you’ll use Laravel’s command-line utility called Artisan. Execute the following command in your terminal window:

php artisan make:model Promotion --migration

This command does two things. First, it creates a model class that you will use later to interact with the database table. Second, it creates a database migration file that you use to, among other things, define the promotions table schema.

Navigate to your project’s database\migrations\ folder and open the file that ends with create_promotions_table.php. In it, you will find a class called CreatePromotionsTable, and it contains an up() method that looks like the following:

public function up()
{
  Schema::create('promotions', function (Blueprint $table) {
    $table->id();
    $table->timestamps();
  });
}

This code uses the Schema::create() method to create the promotions table, and you’ll use the $table object to define the table’s columns. For this article, your author decided to implement a solution that allows for multiple promotions based upon the date, the visitor’s region, and/or the visitor’s city. Therefore, you can define promotions that are limited to visitors within a specific city, a specific region (eg: state, province, etc), or are not limited to anyone at all.

To account for these different variables, the promotions table needs the following columns:

• start date
• end date
• region
• city
• discount amount

The following code defines those columns using Laravel’s table builder syntax:

public function up()
{
  Schema::create('promotions', function (Blueprint $table) {
    $table->id();
    $table->date('start_date');
    $table->date('end_date');
    $table->string('region_code');
    $table->string('city', 100)->nullable();
    $table->decimal('discount', 2, 2);
    $table->timestamps();
  });
}

Of course, you need some data to work with. You can manually insert records using your database management tool of choice, or you can use Laravel’s seeding capabilities to quickly populate the promotions table.

In a large application, you would want to create a custom seeder class specifically for your promotions table. In this case, however, you can simply add code to the default database seeder. Open the database\seeders\DatabaseSeeder.php file and add the following use statements:

use Illuminate\Support\Facades\DB;
use Carbon\Carbon;

The first use statement imports the DB facade, which provides the methods you’ll need to insert records into the promotions table. The second statement imports Carbon, a simple API extension for working with date and time.

Next, modify the run() method to insert records into the promotions table, like this:

public function run()
{
    // User::factory(10)->create();
  DB::table('promotions')->insert([
    'start_date' => Carbon::create('2020', '07', '01'),
    'end_date' => Carbon::create('2020', '10', '31'),
    'region_code' => '*',
    'city' => null,
    'discount' => .10
  ]);

  DB::table('promotions')->insert([
    'start_date' => Carbon::create('2020', '08', '01'),
    'end_date' => Carbon::create('2020', '9', '30'),
    'region_code' => 'TX',
    'city' => null,
    'discount' => .25
  ]);

  DB::table('promotions')->insert([
    'start_date' => Carbon::create('2020', '08', '01'),
    'end_date' => Carbon::create('2020', '9', '30'),
    'region_code' => 'TX',
    'city' => 'Houston',
    'discount' => .30
  ]);
}

The highlighted code in this example inserts three records into the promotions table. Naturally, make sure your start and end dates are suitable for purposes, and it is also important that any city you specify actually resides within the specified region.

Notice line 7 in the above code. The region_code column contains a value of ‘*’. This, in conjunction with a null city, is a “catch all” promotion. If a visitor is not in a specified city or region, then they can take advantage of the “catch all” promotion as long as they visit within the start and end date.

Your next step is to migrate and seed the database, and you can do so by executing the following commands in the terminal:

php artisan migrate
php artisan db:seed

As you’ve already guessed, the first command executes the migration and actually creates the promotions table. The second command executes the run() method in the DatabaseSeeder class, therefore inserting the specified records into the table.

Now that you have a some test data in an actual database table, you can focus on writing the service class that will retrieve the most appropriate promotion for a given visitor.

Writing the Service Class

As its name implies, the app folder in a Laravel project typically contains the application code. With that in mind, create a subdirectory in app called Services. Then create a new file called PromotionService.php and start the file with the following lines of code:

namespace App\Services;

use App\Models\Promotion;
use Ipdata\ApiClient\Ipdata;
use Symfony\Component\HttpClient\Psr18Client;
use Nyholm\Psr7\Factory\Psr17Factory;

class PromotionService {
    // more code to come
}

The first line in this code defines the App\Services namespace. While not entirely necessary, using namespaces is a fantastic way of organizing your application’s code.

The next four lines of code are use  statements that import the following classes:

• Promotion—the model class you will use to query the database.
• Ipdata—the client class for interacting with the ipdata API.
• Psr18Client and Psr17Factory—dependencies needed to create an Ipdata client object.

Next, define a private method called createClient(), as shown in the following code:

class PromotionService {
  private function createClient() {
    $httpClient = new Psr18Client();
    $psr17Factory = new Psr17Factory();
    return new Ipdata('<your_ip_key>', $httpClient, $psr17Factory);
  }
  // more code here
}

Creating an Ipdata client is very straight forward. Simply “new up” the class’ constructor and pass three arguments in the following order: your API key, a Psr18Client object, and an instance of Psr17Factory.

The next method is called getPromotion(),  and its purpose is to query the promotions table , process the query results, and return the appropriate promotion for the provided region code and city. Start with the following code:

class PromotionService {
  private function createClient() {
    $httpClient = new Psr18Client();
    $psr17Factory = new Psr17Factory();
    return new Ipdata('<your_ip_key>', $httpClient, $psr17Factory);
  }
  
  private function getPromotion($regionCode, $city) {
    $now = date('Y-m-d');
    // more code here
  }
}

This code defines the method and creates a variable to contain the current date. Remember, in addition to a visitor’s location, promotions also depend upon the date; so, you’ll use this $now variable to find a current promotion.

In plain SQL, the basic query looks like the following:

SELECT * 
FROM   promotions 
WHERE  start_date < $now
       AND end_date > $now

This query simply selects the promotions that have a start date before, and an end date after, the current date (represented by $now). You also want to find promotions that match your visitor’s region, but remember that a promotion may be globally available—a “catch all” promotion. Therefore, you need to find any promotion where the region_code column is either an asterisk (*) or the supplied region code (represented by $regionCode below). That can be done with the following:

SELECT * 
FROM   promotions 
WHERE  start_date < $now
       AND end_date > $now
       AND region_code IN ( '*', $regionCode )

The city slightly complicates things. The city column can be NULL, indicating that the promotion applies to anyone in the specified region, but it can also contain an actual city name.

By themselves, city names are not unique. For example, what region/country comes to mind when you think of the city name of “Paris”? Like your author, you probably thought of France—unless you happen to live in or near one of the 23+ cities in the USA that are also named Paris.

A city name is only unique when it is combined with a region. I point this out because unlike the region_code column, you cannot use an IN clause when comparing values for the city. Instead, you have to check if city is null or if city matches the provided $city, and you have to group those comparisons together. In case that’s clear as mud, hopefully the following SQL will clarify:

SELECT * 
FROM   promotions 
WHERE  start_date < $now
       AND end_date > $now
       AND region_code IN ( '*', $regionCode )
       AND ( city IS NULL 
              OR city = $city )

This is the query you need to execute in order to find all of the possible promotions for a visitor. In Laravel applications, you can execute raw SQL statements with no problem. However, you can also use model classes, like your Promotion class, and the query builder syntax to execute statements. The Laravel equivalent to the above SQL is the following:

Promotion::where('start_date', '<', $now)
  ->where('end_date', '>', $now)
  ->whereIn('region_code', ['*', $regionCode])
  ->where(function($query) use ($city) {
      $query->whereNull('city')
          ->orWhere('city', $city);
  })
  ->get();

Therefore, the first few lines of the getPromotion() method looks like this:

private function getPromotion($regionCode, $city) {
  $now = date('Y-m-d');
  $promotions = Promotion::where('start_date', '<', $now)
    ->where('end_date', '>', $now)
    ->whereIn('region_code', ['*', $regionCode])
    ->where(function($query) use ($city) {
      $query->whereNull('city')
        ->orWhere('city', $city);
      })
    ->get();
  // more code here
}

But remember: this finds all of the possible promotions for the visitor. So, you’ll need to process the results in order to find the promotion that best fits the visitor’s location. In the code above, the $promotions variable contains a special enumerable object called a Collection. You could iterate over the collection with a normal for or foreach loop. However, a Collection exposes many useful methods that you can use to find a specific record.

In this case, you want to find a single record that matches the visitor’s city or region code, and you can easily find that object by using the first() method, like this:

private function getPromotion($regionCode, $city) {
  $now = date('Y-m-d');
  $promotions = Promotion::where('start_date', '<', $now)
    ->where('end_date', '>', $now)
    ->whereIn('region_code', ['*', $regionCode])
    ->where(function($query) use ($city) {
      $query->whereNull('city')
        ->orWhere('city', $city);
      })
    ->get();

  return $promotions->first(function($record) use ($city, $regionCode) {
    return 
      $record->city == $city || 
      $record->region_code == $regionCode || 
      $record->region_code == '*';
  });
}

A collection’s first() method accepts a closure that executes on every item within the collection. This closure first tries to match the record’s city with the visitor’s city. If there’s a match, the city-matching record is returned. If a city-match is not found, the closure tries to match a record’s region code with the visitor. Once again, if a match is found, the matching record is returned. If a record cannot be matched by city or region code, then the closure attempts to find a “catch all” record. If no match is found, it returns null.

The last method in the PromotionService class brings everything together. It’s called getPromotionFor(), and it will create the ipdata client, lookup the provided IP address, and query the database using the resulting location data. The following code is the complete PromotionService class:

class PromotionService {
  public function getPromotionFor($ipAddress) {
      $client = $this->createClient();
      $result = $client->lookup($ipAddress);
      return $this->getPromotion($result['city'], $result['region_code']);
  }

  private function getPromotion($city, $regionCode) {
      $now = date('Y-m-d');
      $promotions = Promotion::where('start_date', '<', $now)
          ->where('end_date', '>', $now)
          ->whereIn('region_code', ['*', $regionCode])
          ->where(function($query) use ($city) {
              $query->whereNull('city')
                  ->orWhere('city', $city);
          })
          ->get();
      var_dump($promotions);
      $selectedPromo = null;
      return $promotions->first(function($item) use ($city, $regionCode) {
          return 
              $item->city == $city || 
              $item->region_code == $regionCode || 
              $item->region_code == '*';
      });
  }

  private function createClient() {
      $httpClient = new Psr18Client();
      $psr17Factory = new Psr17Factory();
      return new Ipdata('test', $httpClient, $psr17Factory);
  }
}

Using the PromotionService Class

Now that you completed the PromotionService class, all you have to do is use it. Open the project’s routes\web.php file and add the following use statement:

use App\Services\PromotionService;

Next, modify the single route defined in the file. You want to create a service object, use its getPromotionFor() method, and pass the result to the view, like this:

Route::get('/', function () {
  $service = new PromotionService();
  $promotion = $service->getPromotionFor(\request()->ip());
  return view('welcome', ['promotion' => $promotion]);
});

The highlighted code in this code are the additions you need to make. Save the file, and then open the welcome.blade.php file located in your project’s resources\views\ directory. Directly after the opening <body> tag, add the following code:

@if (isset($promotion))
    <h2>Grats! You get a {{ $promotion->discount }} discount at checkout!</h2>
@endif

This code checks to see if the visitor receives a promotion (remember that if no promotion is found, $promotion is null), and if so, it displays a message indicating the discount amount.

Conclusion

The web revolutionized the way companies conduct business. It is the “great equalizer” because small and local businesses have the same potential customer base as large corporations. But that doesn’t mean small businesses have to eschew their local base in favor gaining global customers. It’s quite the opposite. By leveraging modern tools like ipdata’s location API, local businesses can cater to their most loyal customers, strengthening the relationship within the community.

What is a proxy?

A proxy is a simple server which forwards requests on your behalf. For example, if you’re using a proxy to access google.com, your request will first be sent to the proxy server, which will then in turn make the actual request to google.com, and return the result back to you.

To help explain it, let’s look at a simplified diagram of how your computer connects to a website.

And here’s how a proxy fits in…

How is a proxy different from a VPN?

This all looks very similar to a VPN, as we explored in our Tor Detection post, but there are some key differences.

When you connect to a VPN, it establishes a secure, encrypted connection between your computer and the Internet. Your computer is connected to a remote and Virtual Private Network, so all Internet traffic will be sent over this connection, regardless of whether it’s from your web browser, a native application, or even your operating system checking for updates. Because of this, the VPNs are much heavier and have the potential to add more latency to your traffic. However, they’re more fully-functioning and generally more secure.

Traffic from VPNs can look identical to proxy traffic, so from a detection perspective there’s no difference – the IP address of the VPN or proxy can be looked up against a database or service, such as ipdata.

Underlying technology

The term “proxy” covers any server that forwards data on behalf of another user or server. Due to the broad coverage, there are many different types and styles of proxies, but most of these will use either HTTP or SOCKS as an underlying technology.

HTTP Proxies

An HTTP proxy is one of the simplest types, and can be written in just a few lines of code. In Node.js, the module node-http-proxy makes the process very simple.

const httpProxy = require('http-proxy');
httpProxy.createProxyServer({ target:'https://dogtreats.com' }).listen(8000);

Now, if we run this, any requests to port 8000 will be forwarded on to https://dogtreats.com and look as if they originated from the server where this is running.

SOCKS Proxies

A SOCKS proxy runs at a lower-level than a HTTP proxy, making it far more versatile. SOCKS can forward TCP and UDP connections and is therefore more commonly used than HTTP proxies. A SOCKS proxy can also be run over SSH using OpenSSH, allowing users to securely connect to servers.

Tor is a system which effectively links multiple SOCKS proxies together to provide strong anonymity, at the cost of speed and some convenience. Read more about Tor detection here.

Proxy Categories

Any of these proxy types can use any of the proxy technologies, such as SOCKS. A proxy can also fall into multiple categories!

Transparent proxies simply pass on the request from the user to the destination, without making any modifications at all. Non-transparent proxies make some modification to the request, such as adding HTTP headers. Some proxies add an [x-forwarded-for](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For) header which includes the original IP address of the user, this would be an example of a non-transparent proxy.

Open proxies are simply proxies which are accessibly by anyone on the internet, sometimes by mistake. Due to their open nature, they’re often used to provide anonymity and are frequently used for malicious activity. Using open proxies is extremely dangerous, unless you totally trust the proxy service you’re connecting to, as they may be snooping on all your internet usage.

Corporate proxies are proxies run by corporations, usually hosted on their own servers. Many companies use proxies to control access to the internet and their networks from their employees.

Private proxies are very similar to open proxies, but they require some kind of registration and usually promise to be more secure than open proxies.

Detecting proxies

It’d be impossible to detect all proxy servers, but some can be detected. Proxy providers continually change their IP addresses to try to avoid detection.

There are multiple lists of known proxy IP addresses available online, and ipdata combines many of them, along with proprietary lists to detect the larger proxies.

Additionally, proxies are reasonably likely to be hosted by a cloud hosting provider, such as AWS or OVH. Traffic from hosting providers can usually be detected using the IP address, but of course there are sometimes legitimate reasons for a hosting provider to be calling your service – especially if you’re running an API.

Here’s an example ipdata response for a proxy:

curl https://api.ipdata.co/54.39.133.108?api-key=test


{
    ip: "54.39.133.108",
    is_eu: false,
    city: null,
    region: null,
    region_code: null,
    country_name: "Canada",
    country_code: "CA",
    continent_name: "North America",
    continent_code: "NA",
    latitude: 43.6319,
    longitude: -79.3716,
    postal: null,
    calling_code: "1",
    flag: "https://ipdata.co/flags/ca.png",
    emoji_flag: "🇨🇦",
    emoji_unicode: "U+1F1E8 U+1F1E6",
    asn: {
        asn: "AS16276",
        name: "OVH SAS",
        domain: "ovh.com",
        route: "54.39.0.0/16",
        type: "hosting"
    },
    languages: [
        {
            name: "English",
            native: "English"
        },
        {
            name: "French",
            native: "Français"
        }
    ],
    currency: {
        name: "Canadian Dollar",
        code: "CAD",
        symbol: "CA$",
        native: "$",
        plural: "Canadian dollars"
    },
    time_zone: {
        name: "America/Toronto",
        abbr: "EDT",
        offset: "-0400",
        is_dst: true,
        current_time: "2020-09-04T12:42:43.132643-04:00"
    },
    threat: {
        is_tor: false,
        is_proxy: false,
        is_anonymous: false,
        is_known_attacker: false,
        is_known_abuser: false,
        is_threat: false,
        is_bogon: false
    }
}

Here, we can see ipdata hasn’t currently got this IP address on a known proxy list (threat.is_proxy ), but the request is coming from a hosting provider ( asn.type).

Blocking proxies

Due to the difficulty in reliably detecting proxies, it’s usually not worth spending much time trying to block proxies. However, some proxies will be caught by threat.is_proxy and generally, ipdata will help protect your site from threats if you block all requests where threat.is_threat == true.

There are also many, totally legitimate, use-cases for proxies. Corporate proxies are common and blocking legitimate traffic from these companies could be costly!

Due to a large amount of illegal use, some internet providers and websites block access from Tor. We’ll explore how to detect Tor traffic, how to block it, and analyse whether blocking is a good approach.

How does Tor work?

Tor uses a process called “onion routing”, and its name was originally an acronym for “The Onion Router”. Onion routing was developed by the U.S. Naval Research Lab and is named “onion” due to the layers of privacy and encryption.

To help explain it, let’s look at a simplified diagram of how your computer connects to a website.

When you open a website, your real IP address is exposed, which can be used to determine your location, ISP and much more. Using HTTPS means your connection is encrypted, so your data is safe from interception, snooping, or tampering. However, your ISP can see which website you’re connecting to, or even change it, which means they are often asked to block websites, such as The Pirate Bay.

To get around this, some users will use a VPN, which sits between the connection between your computer and the website, helping to hide your real IP and identity.

Tor is like chaining together multiple VPNs at the same time. Your laptop connects to a Tor entry node, which in turn connects to a Tor relay. Your connection will pass through multiple relays, with each step adding another layer of security and re-encrypting the data.

A Tor exit node ends up connecting to the website you want to visit. Only the Tor entry node knows your real IP address.

How can I detect and block Tor traffic?

To detect Tor traffic, we need a list of all of the known IPs of Tor exit nodes. We don’t need any information about the relays or the entry nodes, as they’ll never connect to our website.

Tor publishes an official list of exit node IP address. There are fewer than 2000 IPs in that list, so it’s not too difficult to check if the connecting IP is a known Tor exit node. If we had a simple Express application, we could detect and block this traffic using a middleware.

const fs = require("fs");
const express = require("express");
const app = express();

// Download the official list to a txt file, split it into an array of IP addresses
const torIPs = fs.readFileSync("./tor-exit-nodes.txt").split("\n");

app.use((req, res, next) => {
  const ip = req.connection.remoteAddress;
  // Block any known Tor exit node IP addresses
  if (torIPs.contains(ip)) {
    res.status(403).send('Tor is not allowed');
    return;
  }
  next();
});

// Serve the home page
app.get("/", (req, res) =>
  res.sendFile("./index.html", { root: __dirname })
);

app.listen(8000);

This approach will catch most Tor connections, but there are also some unofficial exit nodes which aren’t on the published list. ipdata maintains an up-to-date list of the official exit nodes, and combines that with a proprietary list of unofficial Tor IPs. The data is aggregated every 15 minutes and updates every hour, meaning you’ll get the most accurate Tor detection possible.

Let’s update our application to use ipdata’s API instead.

const express = require("express");
const app = express();
const axios = require('axios');

// Get an ipdata API Key from here: https://ipdata.co/sign-up.html
const IPDATA_API_KEY = "test";
const getIpData = async (ip) => {
  const response = await axios.get(
    `https://api.ipdata.co/${ip}?api-key=${IPDATA_API_KEY}`
  );
  return response.data;
};

app.use(async (req, res, next) => {
  const ip = req.connection.remoteAddress;
  const ipdata = await getIpData(ip);
  // Block any known Tor traffic
  if (ipdata.threat.is_tor) {
    res.status(403).send('Tor is not allowed');
    return;
  }
  next();
});

// Serve the home page
app.get("/", (req, res) =>
  res.sendFile("./index.html", { root: __dirname })
);

app.listen(8000);

Should I block Tor traffic?

Now that we’ve shown how to block Tor traffic, it’s important to understand that Tor isn’t only used for illegal activities. Tor can be used to bypass the Great Firewall and other regulating systems to allow the free movement of information – currently all versions of Wikipedia are blocked in China. Users are becoming more privacy-conscious, leading to increased usage of VPNs and Tor. So the question should be, “do I want to risk blocking legitimate traffic?”

If you’re running an e-commerce store, you’ll need to be wary of fraud, so you may want to block Tor. It’s unlikely that your store would be blocked by any firewalls, and unless you’re selling sensitive products, blocking Tor should have minimal impact on your real customers.

If you’re running a news site, blocking Tor could restrict people in certain areas from accessing the information they need. There’s minimal risk of fraud, unless you’re taking payments from your site, so it’d usually be best to allow Tor traffic, but monitor it closely for fraudulent activity.

Conclusions

Tor traffic can be blocked using a list of IP addresses, or by using the ipdata API, which also exposes other data such as the user’s location and more detailed threat data. However, it’s worth considering whether blocking all Tor traffic is the right option for you.

What’s your experience with Tor? Have you had issues with visitors hitting your website via Tor? Let us know in the comments!

But with growth comes complexity. Serving a global audience requires a company to adapt their systems for a global economy. That could be a daunting task even five years ago, but it is now relatively simple with services like ipdata and Open Exchange Rates.

In this article, you will build a simple page that converts product prices to the visitor's local currency in real time using the ipdata and Open Exchange Rates APIs.

Getting Started

We need API keys before we can do anything. If you don't already have an ipdata account, sign up for a free account. Next, create an account at Open Exchange Rates. Their pricing page does not prominently display their free plan, but don't fear—it's forever free.

For this article, we'll build a simple web page that uses plain ‘ol JavaScript to display a list of products and their respective prices in the visitor's currency. The only library you'll use in this project is the money.js library for converting our product price to the visitor's local currency.

The overall process is simple:

• fetch product data (it's just a static .json file to emulate a local service)
• fetch the visitor's location data from ipdata's API
• fetch the currency exchange information from Open Exchange Rates
• display the product data with the converted currency values

Because the page issues HTTP requests, it must be served from an HTTP server. Any HTTP server will do. If you don't have easy access to one, the (lite-server NPM package)[https://github.com/johnpapa/lite-server] can get you up and running with a simple HTTP server with no configuration.

Building the Product Page

You'll emulate the products “service” with a simple JSON file. Create a new file called data.json, and type the following:

[
    { "name" : "Guitar", "price" : 1500 },
    { "name" : "Piano", "price" : 5000 },
    { "name" : "Cello", "price" : 15000 }
]

Next, create an HTML file with the following markup:

<html lang="en">
<!DOCTYPE html>
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Musical Instruments</title>
</head>
<body>


    <script src="https://openexchangerates.github.io/money.js/money.min.js"></script>
</body>
</html>

As you can see, the only dependency in this page is the Open Exchange Rate’s money.js library. Of course, you could use the respective JavaScript libraries for the ipdata and Open Exchange Rate services, but you will directly interact with those web APIs using fetch().

Now you need someplace to display the product list. You could build everything using JavaScript, but doing so without a UI-focused library is an exercise in tedium. Instead, add the following table to the body of the page. Later, you will programmatically insert rows into the table.

<table id="product-table">
    <thead>
        <tr>
            <th>Name</th>
            <th>Price</th>
        </tr>
    </thead>
    <tbody></tbody>
</table>

Fetching Data

Now it’s time to fetch your data. For the sake of simplicity, use async/await when calling fetch(). All modern browsers support async/await, and it saves you from the Promise callback hell that is associated with multiple asynchronous operations.

Begin by defining an immediately invoked asynchronous function, as shown in the following code:

(async function() {
    // all of our JavaScript code here
})();

The Fetch API is a huge improvement over XMLHttpRequest, but it can still be tedious to use when working with HTTP resources that return JSON. However, a helper function that fetches data and returns the resulting parsed JavaScript object can be beneficial. The following fetchJson() function does just that:

async function fetchJson(url) {
    let res = await fetch(url);
    return await res.json();
}

It simply fetches the provided URL and uses the response object’s json() method to transform the JSON structure into a usable JavaScript object.

Next, fetch your data with the following three lines:

let products = await fetchJson('data.json');
let location = await fetchJson('https://api.ipdata.co?api-key=<YOUR_API_KEY>&fields=currency');
let exchange = await fetchJson('https://openexchangerates.org/api/latest.json?app_id=<YOUR_API_KEY>');

This code fetches the product data, the visitor’s location information, and the latest currency exchange information. This code is straight-forward, but let's focus on the ipdata request.

First, ipdata’s API returns location-based information about the device that issued the request. In this case, that is the user’s browser, and that is exactly what we want. Remember, you are converting product prices to the user’s currency based upon their current location, and you automatically get that with just issuing a "blank" request.

Second, you don’t need to know everything about the user’s location; the only thing you really care about is the currency information. Therefore, use the fields query parameter to tell the API that you only want the currency object in the returned JSON structure. This saves bandwidth and resources on the device.

Displaying the Products

Before using the money.js library, you need to initialize its base and rates information. It’s just takes two lines of code using data from the Open Exchange Rates API:

fx.base = exchange.base;
fx.rates = exchange.rates;

Next, get a reference to the <tbody/> in the table and start iterating over the products, like this:

let tbody = document.getElementById('product-table').tBodies[0];

products.forEach(function(p) {
   // more code here    
});

The goal here is to use the tbody object to create a new row for each iteration through the products array. While that sounds like a lot of tedious DOM code, it’s actually quite simple. The API for working with tables and their related objects are very intuitive, as shown in the following example:

products.forEach(function(p) {
    let row = tbody.insertRow();

    let nameCell = row.insertCell();
    let priceCell = row.insertCell();

    nameCell.innerHTML = p.name;
    // price cell here    
});

In this code, you create a new row using insertRow(). This automatically creates the resulting <tr/> element and appends it to the <tbody/> element. Using the newly created row object, you can then create the cells for that row using the insertCell() method. This method automatically creates the necessary <td/> element and appends it to the row. Don’t you wish the rest of the DOM API is as easy to use?

The final piece of this puzzle is the price. The money.js library provides multiple ways to convert values from one currency to another. For this project, use the fluent approach, shown below:

fx(price).from(‘FROM_CURRENCY_CODE’).to(‘TO_CURRENCY_CODE’)

For this project, the user is whoever is viewing the page (you, in this case). So, if you’re going to convert the price to your own currency, FROM_CURRENCY_CODE needs to be a currency that is not yours. This example uses the euro (code: EUR) as the starting currency, but feel free to use whichever currency you want. To plug in your values, the code will look like the following:

fx(p.price).from('EUR').to(location.currency.code).toFixed(2);

Conversions rarely result in clean numbers; chances are very good that the resulting amount will have more than two decimal places. Therefore, call toFixed() to constrain the result to two decimal places.

But converting the price is only part of the solution; you also need to include the correct currency symbol. Thankfully, you already have that from the ipdata API. Just use the currency.symbol property and concatenate it with the conversion result, like this:

priceCell.innerHTML = location.currency.symbol + '' + 
    fx(p.price).from('EUR').to(location.currency.code).toFixed(2);

So the JavaScript code in its entirety is the following:

(async function() {
    async function fetchJson(url) {
        let res = await fetch(url);

        return await res.json();
    }

    let products = await fetchJson('data.json');
    let location = await fetchJson('https://api.ipdata.co?api-key=<YOUR_API_KEY>&fields=currency');
    let exchange = await fetchJson('https://openexchangerates.org/api/latest.json?app_id=<YOUR_API_KEY>');

    fx.base = exchange.base;
    fx.rates = exchange.rates;

    let tbody = document.getElementById('product-table').tBodies[0];

    products.forEach(function(p) {
        let row = tbody.insertRow();

        let nameCell = row.insertCell();
        let priceCell = row.insertCell();

        nameCell.innerHTML = p.name;
        priceCell.innerHTML = location.currency.symbol + '' + 
            fx(p.price).from('EUR').to(location.currency.code).toFixed(2);
    });
})();

Save the file and request it from your HTTP server. You should see the products displayed in the page with the prices converted to your currency.

Personalizing Experiences

The web is a global community, and it can certainly feel cold and inhuman, but using location data to personalize your user’s experiences goes a long way toward developing a relationship with your users. Converting product and service prices in real-time is just one way that location data can help us build meaningful experiences. No matter what kind of site you’re building, use location data to help your users feel at home. They’ll more than likely come back.

Even with a well-designed form, scrolling through all the calling codes (around 243, depending on which source you use) is difficult and slow. If you’re able to search the list, it’s still frustrating, is it +44 for the Isle of Man, Guernsey, United Kingdom, or Jersey? Does it matter? Will selecting the wrong country break everything? All these questions can easily be answered without any user input and solved using some simple JavaScript.

How NOT to do it

First, let’s take a look at how many websites implement their country code selection…

Scrolling through hundreds of options is not good enough for your users. Let’s give them something better.

How to do it

There’s an excellent library called [intl-tel-input](https://github.com/jackocnr/intl-tel-input) which can provide us with an easy-to-use and pretty input for collecting phone numbers with their calling code. Let’s add it to our page and see how much better it is.

<!-- Load the intl-tel-input styles -->
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/intl-tel-input/17.0.3/css/intlTelInput.min.css" />

<!-- This is our phone number input -->
<input type="tel" id="phone" />

<!-- Load the intl-tel-input script -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/intl-tel-input/17.0.3/js/intlTelInput.min.js"></script>

<!-- Initialise the library -->
<script>
  var input = document.querySelector("#phone");
  window.intlTelInput(input, {
    // utilsScript is useful for providing validation and pretty formatting
    utilsScript: "https://cdnjs.cloudflare.com/ajax/libs/intl-tel-input/17.0.3/js/utils.min.js",
  });
</script>

And here are the results, much better!

Unfortunately, due to the large number of countries, there’s still a lot of scrolling involved to find the correct calling code. The library allows users to type in their own calling code, and it’ll update the flag accordingly, but we can still do better.

Auto-select the correct country

Using ipdata, we can look up the user’s country and use that to pre-select the correct calling code. intl-tel-input has an easy option to help make this even simpler.

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/intl-tel-input/17.0.3/css/intlTelInput.min.css" />

<input type="tel" id="phone" />

<script src="https://cdnjs.cloudflare.com/ajax/libs/intl-tel-input/17.0.3/js/intlTelInput.min.js"></script>

<script>
  var input = document.querySelector("#phone");
  window.intlTelInput(input, {
    initialCountry: "auto",
    geoIpLookup: function(success) {
      // Get your api-key at https://ipdata.co/
      fetch("https://api.ipdata.co/?api-key=test")
        .then(function(response) {
          if (!response.ok) return success("");
          return response.json();
        })
        .then(function(ipdata) {
          success(ipdata.country_code);
        });
    },
    utilsScript: "https://cdnjs.cloudflare.com/ajax/libs/intl-tel-input/17.0.3/js/utils.min.js",
  });
</script>

Now, when the library loads, it’ll call our geoIpLookup function and use the country code returned by ipdata. If it’s unable to detect the country for any reason, it’ll fall back to the default country.

Let’s see what happens if we load the page from a location in the Åland Islands, which is located right at the bottom of the list.

Now, we’re greeted with the familiarity of our native flag and allowed to immediately begin entering our phone number without any scrolling. If, for example, the customer is on holiday in a different country, they can still select the correct calling code using the selector.

Conclusions

Making your customers happy should be at the top of your priority list. Giving them slow, clunky, and difficult to use country calling code selector will do exactly the opposite. In just a few lines of code, we’ve been able to totally transform the user experience into a quick and pleasant one.

Location information can be used for much more than collecting phone numbers with country calling codes. Watch this space for more examples of using ipdata.co to improve your users’ form-filling experience!

But of course, you need an IP address before you can use the wealth of information provided by ipdata. In this article, you will learn how to retrieve an IP addresses with JavaScript—both from within the browser and in your server application.

Retrieving Your User’s IP in the Browser

The browser is the OS of the web, and with more processing being handled within the browser, it makes some sense to retrieve the user’s IP address so that you can use it in your client-side code. The problem, however, is that the browser doesn’t expose any network-oriented information about the device.

1. Using ipdata

You must therefore rely upon a web service to provide that information for you. You could write your own service as part of your server-side application, or you can use a third-party solution to retrieve your user’s IP address. ipdata’s web service is a perfect solution for this problem. All you need is an API key (go ahead and sign up for free). For example:

function json(url) {
  return fetch(url).then(res => res.json());
}

let apiKey = 'your_api_key';
json(`https://api.ipdata.co?api-key=${apiKey}`).then(data => {
  console.log(data.ip);
  console.log(data.city);
  console.log(data.country_code);
  // so many more properties
});

This code defines a helper function called json(). It simplifies the process of using fetch() with resources that return a JSON structure, as most APIs do. Next, the json() function is used to send a GET request to ipdata’s default endpoint: http://api.ipdata.co. You can use this endpoint to query any IP address, but if you do not provide one with the request, the web service uses the address of the device that issued the request: the browser in this case.

The web service returns a JSON payload that contains a wealth of information—not just the user’s IP address, but all the locational data that ipdata has about that address. For example: the ISP, city, state, country, continent, etc are provided in the payload. Because of this, you do not need to issue any other HTTP requests because you already have the information you need.

Sometimes, however, you do not need all the extraneous locational information, and ipdata’s API lets you filter the results by including the fields query parameter. For example:

json(`https://api.ipdata.co?api-key=${apiKey}&fields=ip`).then(data => {
  console.log(data.ip);
});

The JSON payload contains only the fields specified by the fields query parameter. So, in this code, the resulting JavaScript object (named data) has a single property called ip.

2. Using Cloudflare

ipdata's web service is very easy to use. However, another (albeit limited) option is CloudFlare’s trace utility found at https://www.cloudflare.com/cdn-cgi/trace. It returns a plain-text set of key/value pairs, and the following shows a condensed example of the format:

h=www.cloudflare.com
ip=178.25.55.75
ts=1605854122.426
visit_scheme=https
uag=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36

As you can see, the output contains an ip key with its associated value, and you can parse it out with a regular expression like this:

function text(url) {
  return fetch(url).then(res => res.text());
}

text('https://www.cloudflare.com/cdn-cgi/trace').then(data => {
  let ipRegex = /[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/
  let ip = data.match(ipRegex)[0];
  console.log(ip);
});

This code defines a text() helper function that is similar to json(), but of course, it reads the response stream as text instead of a JSON structure.

Inside the callback function, you define a regular expression that matches an IPv4 address, and you use that regex to find a match within the response payload. The result gives you the user’s IP address that you can then use in your client-side code.

3. Using Amazon

A similar utility provided by AWS provides another reliable option to query your user’s IP address.

function getIPFromAmazon() {
  fetch("https://checkip.amazonaws.com/").then(res => res.text()).then(data => console.log(data))
}

getIPFromAmazon()

This will give you

178.25.55.75

Retrieving the Client’s IP in the Server (NodeJS)

IP stands for Internet Protocol, and it implements a type of networking called packet switching. It has the concept of hosts (machines), and it defines how data, in the form of packets, are sent between the hosts. In the case of HTTP, there are two hosts: client and server.

When a browser issues an HTTP request to a server, the request is transmitted from the client to the server in the form of IP packets. These packets contain a variety of information, but there are two particularly important things that every packet has: the source host address and the destination host address. Because every packet contains the client address, your server application can retrieve the client’s IP address from the packet.

Every framework, regardless of language and/or platform, has some mechanism for retrieving the client’s IP address. For Node.js applications, there are two such mechanisms:

• the x-forwarded-for header.
• the request.connection.remoteAddress property.

There is a slight issue, however; the x-forwarded-for header can contain multiple, comma-separated IP addresses. Thankfully, the first IP address in the x-forwarded-for header is typically the correct IP address. With that in mind, consider the following code:

const http = require('http');

function requestListener(req, res) {
  let forwarded = req.headers['x-forwarded-for']
  let ip = forwarded ? forwarded.split(/, /)[0] : req.connection.remoteAddress;
  res.writeHead(200);
  res.end(ip);
}

const server = http.createServer(requestListener);
server.listen(3000);

console.log('Server listening at http://localhost:3000');

A lot of this code is boilerplate, so let’s focus on the requestListener() function. The first line of the function retrieves the value of the x-forwarded-for header. It is not always guaranteed to be present. So, in the second line, you check if there is a value, and if so, you split it on a comma (,) and use the first item in the resulting array. If the x-forwarded-for header doesn’t exist, then you use req.connection.remoteAddress as a fallback.

Viola! You have the client’s IP address. It may not be 100% accurate (the end-user could be using a VPN to mask their true IP), but it is more reliable than getting the client’s address through the browser.

Customize Your Content with IPs

As you can see, using JavaScript to find an IP address is relatively simple, and it’s likely that you’ll want to do something with that address. Maybe you’ll want to show different promotions or ads to your users based upon their region, or you may need to convert currency on the fly. Regardless of your needs, ipdata’s web service provides comprehensive locational information based upon your user’s IP address. Go ahead and setup an account for free. It only takes a couple of minutes to get an API key, and you can use it experiment with the API to see what it can do for you.

Internationalization, also called i18n, involves creating a new website or changing your current site in such a way that your site can adapt to a certain local language. Localization, which we touched on in the introduction, is the next step that developers can take to implement one or more languages.

Popular Libraries for Internationalization

Internationalizing sites is easier these days with the many open-source JavaScript libraries available. I’ve found the following libraries to be quite useful.

FormatJS

FormatJS is a collection of JavaScript libraries that helps with number, time, and date formatting. Not only that, but this library also helps you compare language-sensitive strings. The best part is that it includes React-Intl, thus making your React JavaScript app development easier.

AngularI18N

If your app is in Angular, AngularI18N will help you with templates and tools. You can use this library to build helpers that can internationalize and localize your app.

Moment.js

Moment.js helps with streamlining the date-time format. It can also match the format in a specific time zone.

i18next

You can use i18next with Vue, Angular, React, or any framework of your choice. The library helps detect the user’s language. Once done, i18next loads and inserts translations. In addition, it also supports standard features of i18n.

Globalize JS

One of the most popular libraries, Globalize JS uses the Common Locale Data Repository (CLDR), the largest available base of locale data. The library uses CLDR to get the formatting and paring pattern for date-time, number, messages, units, and currencies.

In the next section, I’ll guide you on how to internationalize your JavaScript web app using Globalize. We’ll use Globalize because it uses the largest standard database of locale data.

A Guide to Internationalization Using Globalize JS

As I noted in the previous section, Globalize uses CLDR from the Unicode Consortium. Because it has such a huge base, you don’t have to think about adding any additional modules. Besides, you don’t need to use the entire library. The content is separated from the code. So, if you want to use the locale data for, say, two or three specific zones, you can pick the modules that you need.

Let’s get started. We’ll discuss the installation first and then move on to the features.

How to Install

Your first step is to find out what modules you need. Check out this link, which lets you can check or uncheck the modules. The site will tell you what CLDR JavaScript files and JSON files you need and in what order you need to arrange them.

To install Globalize, you need to have Node.js installed in your system. If you already have Node.js, you can install the entire Globalize library along with its dependencies by running the following command in cmd using npm.

npm install globalize cldr-data --save

Now, let’s find out how to format the date-time, language, and currency using Globalize.

Formatting the Date

First, you need to include the library in your code. You can do that as follows:

var Globalize = require('globalize');

Next, include the CLDR data:

var cldrData = require('cldr-data');

And don’t forget to load the supplemental data:

Globalize.load(cldrData.entireSupplemental());

Finally, load the specific locale data that you need. For this example, I’ll show you how to format your date-time in the Italian format.

Globalize.load(cldrData.entireMainFor('it', 'en-US', 'en-GB'));

Now, let’s format the date in the U.S., UK, and Italian formats:

// Here, we declare today's date
var date = new Date(2020, 2, 15);

// Print date in Italian format
console.log(Globalize('it').formatDate(date));

// Print date in US format
console.log(Globalize('en-US').formatDate(date));

// Print date in UK format
console.log(Globalize('en-GB').formatDate(date));

In the console, you’ll get the following outputs:

"15/2/2020"
"02/15/2020"
"15/02/2020"

Now let’s try formatting a number.

Formatting the Number

To format numbers, first you have to add the CLDR data and supplemental data. Then you have to load the locale data. The steps are the same as before. Now, let’s see the code to format the number.

var number = 1452.67;

// Format in IT standard
console.log(Globalize('it').formatNumber(number));

// Format in US standard
console.log(Globalize('en-US').formatNumber(number));

You’ll get the following output:

"1.452,67"
"1,452.67"

Formatting the Currency

Now let’s try something interesting. How about we format a currency?

After adding the required CLDR and local data, let’s declare a number and format it in U.S. dollars (USD) and euros (EUR).

var value = 456.342;
//Currency formatter for USD

var usdollarFormat = enGlobalize.currencyFormatter('USD');

console.log(usdFormatter(value));

//Currency formatter for EURO

var euroFormat = enGlobalize.currencyFormatter('EUR', {
  style: 'code',
  round: 'ceil'
});

console.log(eurFormatter(value));

You’ll get the following output:

$456.34
456.34 EUR

Formatting the Language

We saved the interesting stuff for last. Date-time and number formatting are simple. But you may wonder how to change the text content of your website. Well, Globalize JS helps you with that as well. But for that, you need to manually declare the translated data in a JSON file.

For example, let’s suppose you need to translate your site’s homepage. We can start with the register and cancel buttons. In the it.json file, write

{
  "it":
      { 
        "register": "Registrati",
        "cancel":"Annulla"
      }
}

Write the other texts and their translated versions in the JSON.

Now it’s time to fetch the data from the JSON and pass them in order to globalize:

const fetchHeaders = new Headers({"accept": "application/json", 'X-Requested-With': 'XMLHttpRequest'});
var locale = it;
fetch("messages/" + locale + ".json", { headers: fetchHeaders }).then(response => {
return response.json();
})
   .then(
         json => {
                  Globalize.loadMessages(json);
                  Globalize.locale(locale);
                  yourRender();
                  }
         );

Here, the locale is the variable where you declare the JSON filename. If the region is Italy (IT), the code wants the file to be present at messages/it.json. Similarly, if you have JSON for other countries, such as France (fr.json) or Japan (jp.json), the code will fetch data from the respective JSONs and translate the texts accordingly. For translating, you need the following code:

function translateButtons() {
	//To translate the Register button
    document.getElementById( "buttonRegister" ).value = Globalize.formatMessage( "register" );
	//To translate the Cancel button
    document.getElementById( "buttonCancel" ).value = Globalize.formatMessage( "cancel" );
}

Globalize JS also helps to format pluralization, relative time, units, and many other kinds of stuff. Select the module you like and start internationalizing your site.

Now that the code is ready, you may wonder how you can find the geolocation of the users accessing your site. In the next section, I’ll introduce you to ipdata, an easy-to-use geolocation API.

How ipdata Can Help With Internationalization

ipdata provides you with a geolocation API. Using that, you can find the location of any user with their IP address, both IPv4 and IPv6. Let’s see how you can do that. Run the following JavaScript code:

var request = new XMLHttpRequest();

//The GET call returns the location of the IP address of the request’s origin
request.open('GET', 'https://api.ipdata.co/?api-key=test');
request.setRequestHeader('Accept', 'application/json');
//Function executes when readystate changes
request.onreadystatechange = function () {

  if (this.readyState === 4) {
    console.log(this.responseText);
  }
};
request.send();

You need to ensure one thing—that all calls are made using https. ipdata doesn’t serve requests on unsecured http. You can use the ipdata API not only in JavaScript but also in C#, Java, Python, PHP, and many other languages. Sign up to get a free API key and start testing.

The best feature of ipdata, apart from location, is that you can also use the API to detect the mobile carrier, currency, and time zone, along with proxies and threats.

Expand Your Limits With Internationalization

While developing a website, remember the “world” in WWW. Your site is meant to be accessed by people from around the world. No matter what kind of site you’re developing, be it e-commerce or a travelogue, target your audience. Find out the locale where your site is meant to be a hit. Start using internationalization in JavaScript and make your site readable by people from different countries with different languages.

This post was written by Arnab Roy Chowdhury. Arnab is a UI developer by profession and a blogging enthusiast. He has strong expertise in the latest UI/UX trends, project methodologies, testing, and scripting.

Later in the post, we’ll discuss what to do when you don’t have ELK available or when a quicker solution is needed. I’m going to assume you already installed, configured, and ran Elasticsearch (and the rest of the ELK stack). If you haven’t, you might want to pause here and come back when everything’s ready.

So, let’s get started!

GeoIP and ELK

As I mentioned before, Elasticsearch is an amazing tool. And it only gets better when paired with Kibana and Logstash. And Logstash is precisely what we’re here to discuss. You probably know that Logstash is what sits between your data and your database for final archiving and analyzing. It can handle an increasing number of sources and feeds—not just Elasticsearch, but also regular databases or email services. It’s a very powerful tool.

Moreover, Logstash can parse, transform, and filter the data. One of those filters happens to be the geoip-filter. So a common use case is to analyze the access logs of a server; you can very easily ingest them (letting Logstash monitor the logs) or feed them directly to the application. The recommended way to start is to download and install the geoip-filter, as recommended by the official documentation.

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip

However, if you run this command on more recent versions, you may find an error as you run that command.

That means that this plugin is actually part of the Elasticsearch distribution, as noted in the docs.

The way the GeoIP filter works is very simple: Elasticsearch ships with a database of IP addresses and their geographic information. So, when the tool parses the IP, it automatically maps it to a geopoint (latitude and longitude), which can later be mapped in the Visualize tab as follows.

Here, I’m using the sample data that ships with Elasticsearch to map the unique visitors to our sample website, sorted by country. If you’re curious enough, you can also jump in the Discover tab and click around your sample data; you’ll find the geolocation field, noted by the world icon right next to it.

The filter basically groups the latitude and longitude together, useful for pinpointing the markers. Easy!

What If I Don’t Have ELK?

The objective remains the same: How can I get visualization or data based on the IP address? This is a feature that’s normally used in websites, but nowadays, being able to obtain geoinformation is more and more important in a growing number of situations. You may serve API endpoints globally, or perhaps you’d like to filter the content shown on your app based on the location of your users. Another use case might come from your marketing department. They’ll probably ask you to count and perform operations and visualizations on the sales funnel for different locations. Either way, you need GeoIP data.

But sometimes using ELK might be a bit of an overkill. In the cases above, setting up an ELK stack might be even more complicated than creating the functionality itself. That can be due to a number of things. Simplicity is the first one that comes to mind. There are a number of tools and services out there that can also serve GeoIP data—normally using a database. Some of them might even be free. As a matter of fact, the Logstash default database for GeoIP data is free and publicly available from the MaxMind website (though previous registration is required).

Now that would require you to find the right database, download it, compile it (the relationships between the files might not work for you), and distribute it across your codebase. Make sure it has all the information you need (the available data changes on a per-database basis).

Did I mention those databases are normally different files? CSV formatted, too. I know, very 1995. Last but not least, you’ve got to check periodically for changes and updates to the data. Once you have all of that, you need to add it to the infrastructure, whether that’s on a separate service in your codebase or you do decide to keep the files. And we haven’t even talked about accessing the data across different languages and environments.

What do you do?

The Alternative

Well, you’re in luck. There’s a service called ipdata that will get you up and running in no time. Let me show you how that works. Performing a simple request to their service (I’ll discuss how to do that in a minute) returns a JSON object with useful information relevant to the geographical location you’re requesting.

{
    "ip": "81.20.41.113",
    "is_eu": true,
    "city": "St Albans",
    "region": "England",
    "region_code": "ENG",
    "country_name": "United Kingdom",
    "country_code": "GB",
    "continent_name": "Europe",
    "continent_code": "EU",
    "latitude": 51.7768,
    "longitude": -0.2843,
    "postal": "AL4",
    "calling_code": "44",
    "flag": "https://ipdata.co/flags/gb.png",
    "emoji_flag": "\ud83c\uddec\ud83c\udde7",
    "emoji_unicode": "U+1F1EC U+1F1E7",
    "asn": {
        "asn": "AS29033",
        "name": "Eckoh UK Limited",
        "domain": "eckoh.com",
        "route": "81.20.32.0/20",
        "type": "isp"
    },
    "languages": [
        {
            "name": "English",
            "native": "English"
        }
    ],
    "currency": {
        "name": "British Pound Sterling",
        "code": "GBP",
        "symbol": "\u00a3",
        "native": "\u00a3",
        "plural": "British pounds sterling"
    },
    "time_zone": {
        "name": "Europe/London",
        "abbr": "BST",
        "offset": "+0100",
        "is_dst": true,
        "current_time": "2020-05-21T18:07:37.049274+01:00"
    },
    "threat": {
        "is_tor": false,
        "is_proxy": false,
        "is_anonymous": false,
        "is_known_attacker": false,
        "is_known_abuser": false,
        "is_threat": false,
        "is_bogon": false
    }
}

Isn’t that amazing? The response times are equally as amazing as the information presented. The service runs globally on AWS, which allows them to respond in an average of 60 ms. That’s without the use of cache, too. Signing up for their service is very fast. I requested aFree Tier API Key and received a response very quickly to activate my account.

So let’s jump ahead. At this point, I’m sure you already have a number of use cases in mind. However, if you need ideas, the welcome email they send you includes a number of test calls. How’s that for inspiration?

Oh, by the way, did I mention ipdata also handles IPv6? That may not seem like much. It may not even ring a bell, but believe me. It’s a thing. You want your service to be able to handle both versions of the protocol.

So How Does It Work?

Pretty easily. Here’s a sample of a simple curl request. Use the following command:

curl https://api.ipdata.co/8.8.8.8?api-key=API_KEY

And very quickly, the results show up.

Wrapping Up

At its most basic functionality, ELK is a search engine, capable of transforming and displaying data much faster than a regular database manager can. The ELK stack is definitely a rising star in the development industry and not everyone can actually use all of its functionality. So, if you’re here, it means that you’re beginning to use it or perhaps more of a seasoned developer. Either way, you should feel lucky. After all, everyone can write MySQL … right?

However, the more tools you have available as a developer, the more prepared you’ll be when the next challenge arrives. Acquiring GeoIP information will always be a staple resource in commercial and corporate codebases. Not to mention how quickly it can spread to lower-scale environments due to the increased value of customizing content for your users. The possibilities are truly endless. So, if you’re using centralized logging while visualizing global threats in a real-time dashboard, ELK is your friend.

For anything else that requires quick turnarounds and a solution that works on all modern programming languages and platforms, you require something different. Then ipdata has your back.

This post was written by Guillermo Salazar. Guillermo is a solutions architect with over 10 years of experience across a number of different industries. While his experience is based mostly in the web environment, he's recently started to expand his horizons to data science and cybersecurity.

What are ASNs?

Autonomous System Numbers are used by all services which look up company details by IP address. They identify the organisation or Internet Service Provider which controls a set of IP addresses. They’re used in BGP to route information to the correct network. Unless you’re a network administrator, you don’t need to understand the specifics, but they’re essential for looking up company details.

By keeping an up-to-date list of ASNs, it’s possible to look up a specific IP address and establish which organisation owns it. In most cases, we aren’t interested in situations where an ISP, such as Comcast or Virgin Media, owns the IP address, but the information can still be valuable to us. Maintaining this list is a lot of work, so most people opt to use a provider, like ipdata, to look up ASN details, along with other IP address metadata, such as location.

Looking up company details

ipdata provides a simple API to look up ASN details. Let’s try it out with a few examples:

Microsoft Cloud hosting IP

curl "https://api.ipdata.co/13.107.6.152/asn?api-key=test"
{
    "asn": "AS8068",
    "name": "Microsoft Corporation",
    "domain": "microsoft.com",
    "route": "13.107.6.0/24",
    "type": "hosting"
}

AT&T IP

curl "https://api.ipdata.co/32.100.100.100/asn?api-key=test"
{
    "asn": "AS7018",
    "name": "AT&T Services, Inc.",
    "domain": "att.com",
    "route": "32.96.0.0/13",
    "type": "isp"
}

UK Government (House of Commons) IP

curl "https://api.ipdata.co/194.60.38.10/asn?api-key=test"
{
    "asn": "AS29214",
    "name": "THE HOUSE OF COMMONS",
    "domain": "parliament.uk",
    "route": "194.60.38.0/24",
    "type": "isp"
}

As you can see, each IP returns the ASN associated with the IP, along with additional data about the owner. We’re able to know when our website is being used by UK Parliament or many other organisations, businesses, or governments.

ASN Type

A unique feature of ipdata is the ASN type. We attempt to match each ASN to the type of entity which owns it. The possible types are as follows:

TypeDescriptionhostingBelonging to a datacenter (we have mapped 200M+ hosting IP addresses)ispBelonging to ISP IP space (we have mapped 1B+ ISP IPs)eduEducational institutionsgovOne of ~25,000 government agencies worldwidemilMilitary organizationsbusinessEnd-user organizations

We’re mostly interested in the business type for lead prioritisation. However, due to the large number of ASNs, there will always be small corrections or mistakes, meaning the type may be wrong for some IPs. In the examples above, the House of Commons IP should likely be mapped to gov, but is reported as an isp, as that’s how it’s represented in public records. For this reason, the type should be used as an indicator, rather than a fact.

The hosting IPs are accurate and include most cloud hosting providers. This can be useful if you want to block crawlers, scrapers and bots from hosting providers. The first example shows the results for a request from one of Microsoft’s backend services.

Enriching a “Contact Us” form with company details

Let’s build a quick practical example. We have a “Contact Us” form on our website, allowing potential customers to get in touch. We want to include the company details of the user contacting us, without asking them to fill in additional fields.

<form action="/contact" method="POST">
  <input name="email" type="email" placeholder="Email address" />
  <textarea name="message">Your message here...</textarea>
  <input type="submit" />
</form>

Here’s a Node.js/Express application to handle serving the contact form, and handle contact messages, emailing our contact@dogtreats.com email with the details.

const express = require("express");
const app = express();
const axios = require("axios");
const nodemailer = require("nodemailer");
const bodyParser = require("body-parser");
const email = nodemailer.createTransport({
  // https://nodemailer.com
  sendmail: true,
});

// Get an ipdata API Key from here: https://ipdata.co/sign-up.html
const IPDATA_API_KEY = "test";
const getIpData = async (ip) => {
  ip = "81.107.235.77";
  const response = await axios.get(
    `https://api.ipdata.co/${ip}?api-key=${IPDATA_API_KEY}`
  );
  return response.data;
};

// Serve the contact page
app.get("/contact", (req, res) =>
  res.sendFile("./contact.html", { root: __dirname })
);

// Handle a contact request
app.post("/contact", bodyParser.urlencoded(), async (req, res) => {
  const ip = req.connection.remoteAddress;
  const ipdata = await getIpData(ip);
  const message = `From: ${req.body.email}
Message:
${req.body.message}
Company details:
Name: ${ipdata.asn.name}
Domain: ${ipdata.asn.domain}
Type: ${ipdata.asn.type}
`;
  await email.sendMail({
    from: req.body.email,
    to: "contact@dogtreats.com",
    subject: "Contact Us Message",
    text: message,
  });
  res.status(200).send("Thanks for your message!");
});

app.listen(8000);

The email sent to contact@dogtreats.com looks like this:

Of course, this could all use some styling to make it more attractive! However, the functionality is all there and in this example, we’re able to see when someone from within the House of Commons is asking for some dog treats.

Conclusions

With a very small amount of code, we’ve been able to enrich our Contact Us form with company details with just an IP address. Using the ipdata API, we can also include other useful information, such as their location, language and currency.

The uses of company details go beyond a simple contact us form, allowing advanced lead prioritisation and much more. Let us know how you’re using it in the comments below!

Use a cloud-managed service

As we’re concerned about security, we should offload as much of the blocking as possible onto a third party or managed service. This is important as unless you’re an expert in responding to DDoS attacks, handling it yourself could result in very high costs and user-impact.
Ideally, we’ll block requests using the first resource which requests will hit. If we allow potentially malicious requests to pass through multiple resources, or even hit our application servers, our service will be far more vulnerable to attacks, such as Denial-of-Service.
The first resource is usually a CDN, so let’s look at how we can block requests using 4 leading CDNs - CloudFront, CloudFlare, Fastly, and Akamai. All major CDN providers should have some kind of in-built geo blocking functionality – if yours is missing it then it’d be worth considering an alternative.

AWS CloudFront

With CloudFront, there’s a geo restriction feature which can be used to whitelist or blacklist certain countries. It’s quite basic, only allowing us to block or allow a list of country codes.

CloudFlare

CloudFlare has an advanced Firewall feature built-in. It’s able to block or allow requests based on multiple different parameters, such as request path, request method, IP address, IP location, and many more.

Fastly allows us to restrict access using Varnish configuration. It provides a large number of geolocation variables, allowing more than just blocking based on country code. To block all non-EU requests, it’s just a few lines of code:

if ( geoip.continent_code != "EU" ) {
  error 403 "Forbidden";
}

Akamai offers a similar solution to CloudFront, named Geo Protection, which also supports regional blocking and US Designated Market Areas.

Web Application Firewalls

A Web Application Firewall (WAF) monitors and blocks web traffic according to rules. CloudFlare’s Firewall is actually a fully-featured Web Application Firewall (WAF), and most hosting providers offer WAF functionality as a core part of their security features. A common use of WAF is to automatically block many common web exploits, such as XSS or SQLi, so it’s often useful regardless of any geo restrictions.
Most WAF solutions can be used with more than just CDNs – for example, AWS’ WAF can be attached to CloudFront distributions, Application Load Balancers and API Gateways – making them far more versatile than the in-built CDN geo blocking. However, they usually cost more than the in-built solutions and they’re more complex to set up.
To set up AWS WAF, first head to the “WAF & Shield” console, then click “Create web ACL”. You’ll then see a wizard which allows you to create a web ACL, associate it with your resources and configure rules.

If you’re using Google Cloud, Google Cloud Shield now supports WAF with geo blocking support. For Azure users, Azure WAF has geomatch rules, currently available in preview. But what can we do if we aren’t using a major cloud provider?

NGINX

NGINX is a very popular load balancer with both a free and a premium (named Plus) offering. Unfortunately, only NGINX Plus supports the GeoIP module to allow geolocation-based blocking. It’s reasonably complex to set up, but the NGINX docs have a great guide.

ModSecurity

ModSecurity is an open-source Web Application Firewall. It was originally a module for use with the Apache HTTP server, but it can now be integrated with NGINX, IIS, and many other solutions. It’s often available when using cPanel’s WebHost Manager (WHM) so, for some services, it’s the only feasible method of blocking traffic. There are two major versions of ModSecurity.
v2 entered support-mode and hasn’t been released since 2018. It only supports geolocation blocking using the MaxMind legacy databases. GeoLite Legacy has been discontinued and is not available for download, and GeoIP legacy is no longer available to new customers. Unfortunately, v2 is still commonly bundled with WHM, rendering it effectively useless for geo blocking.
v3 now supports the GeoIP2 and GeoLite2 databases, allowing us to write a simple rule to block all traffic which doesn’t originate from the EU:

SecGeoLookupDb /path/to/geoip/geolite2-country.mmdb
SecRule REMOTE_ADDR "@geoLookup" "phase:1,t:none,pass,nolog"
SecRule GEO:COUNTRY_CONTINENT "!@streq EU" "phase:1,t:none,log,deny,msg:'Blocking non-EU IP address'"

Application

If security isn’t the main reason for blocking traffic to your site, or you just want to provide additional location-based functionality, you may wish to do a location lookup within your application code. This could be done using the GeoIP databases mentioned above or a managed service, like ipdata. An Express middleware to only allow EU traffic would look as follows:

app.use(async (req, res, next) => {
  const response = await fetch('https://api.ipdata.co?api-key=test');
  const ipdata = await response.json();
  if (ipdata.continent_code !== 'EU') {
    res.status(403).send('Only users from the EU are allowed');
    return;
  }
  // set ipdata on the request object so we can automatically tailor
  // user preferences such as currency
  req.ipdata = ipdata;
  next()
})

Conclusions

If you’re using a popular CDN or cloud provider, one of their in-built solutions will allow blocking website visitors from specific countries. Choosing one of these in-built solutions will be far safer than using a self-hosted or self-managed option, but we’ve also covered geo blocking for NGINX and ModSecurity, as well as within our application code itself. Having geographic information within your application can help to provide advanced features, such as automatically selecting the correct location or for calculating a delivery estimate.