Ok, one last XMPP post for tonight.
I came across an interesting issue with our test XMPP server today. When the server was initially created, it was setup with an x509/SSL certificate that was self signed. That cert expired the other day, and I had to replace it.
Some of the people who were using the server started getting cert warnings while others did not. After a bit of investigating, I found the problem.
There are two ways that an XMPP client can connect to a server.
The first method is simple. You configure your client with your Jabber ID (username@company.com), and define an XMPP server (servername.company.com) to connect to. This is how our early documentation recommeded configuring your clients, but it is not the generally recommended method.
The second method is the preferred one. You define a service record (SRV record) in DNS for _xmpp-client._tcp.company.com which points to your server name. Once your client has your Jabber ID (JID), the client will automatically look up the SRV record, and connect to that service.
Now comes the cert warning. Apparently if you use an SRV record, your x509 cert needs to have a common name (CN) of company.com. If you define a server manually however, your CN needs to be the name of the server.
Our self-signed cert was for company.com. I replaced it by a properly signed cert for servername.company.com, and broke everyone using the preferred configuration method. Seeing the issue, I replaced it with a properly signed cert for company.com, and broke the people following the published documentation.
*sigh*
So now I get to update the documentation and answer the various questions for people. Longer term I'm going to see if its possible to provide certs for both methods. While you can't have two certs on the same port, it may be possible to either use a cert with an altCN, or possible have the SRV port point to a non-standard port so that people who define the server name can connect to the default port number (5222/5223).
Wednesday, July 30, 2008
Managing your own Chat server
Recently for work I've been dealing with trying to build a robust IM server environment for the company I work for. The plan was to allow employees to chat with each other without the conversations ever leaving the company. Our hope was to also tie the system into the various public instant messaging systems (AIM/MSN/YIM) so that we could use the same system to communicate with customers.
After looking around at various offerings, the choices boiled down to a Jabber/XMPP based solution or a SIP/SIMPLE service like Microsoft OCS or Lotus Sametime. Since XMPP has more open source libraries for coding against it, we decided to go the XMPP route. Personally I was happy since the peer to peer nature of the SIP protocol introduces a few problems on our network.
Of course, deciding on XMPP doesn't really narrow things down too much. There are a number of open source XMPP/Jabber servers out there, and quite a few commercial ones as well.
We looked at Jabberd2, eJabberd and Jabber XCP.
As far as I can tell, Jabberd2 and Jabber XCP are very similar in codebase, at least judging by the configuration file format. The thing that the commercial Jabber XCP product provides is a nice web interface for configuring the service, a pair of supported clients (one Windows client, one web based client) as well as commercial support.
eJabberd was impressive. It is an open source application written in Erlang, which I wasn't familiar with. Erlang was a programming language created for distributed computing by Ericsson. On smaller installations, it can handle clustering without any form of external database. It automatically manages synchronization between nodes of the cluster.
Version 2.0 of ejabberd also included a very nice web interface for managing the service, and included an inpressive number of plugins. Configuring the service was very very easy.
At the end of the day, we ended up going with Jabber XCP. The main reason is the one feature that no one else was able to provide. The ability to really tie into the public IM systems, or at least one of them. Any XMPP server can tie into other XMPP services (like Google Talk), but Jabber XCP offered the ability to tie into AIM as well.
The majority of Jabber/XMPP servers offer transports for the various IM services. What these transports do is essentially allow you to log into your own AOL/MSN/etc account from within your IM client. So I may be myname@company.com within my company, but I would be logging into my gdfuego account on AIM.
Jabber XCP has a plugin (for additional per user cost), which allows you to actually use the same myname@company.com address within AIM itself. Unfortuantely MSN and YIM aren't an option at this time.
Right now I'm getting close to turning our new server live, and I've been hitting a number of snags, but that'll be a story for another day.
After looking around at various offerings, the choices boiled down to a Jabber/XMPP based solution or a SIP/SIMPLE service like Microsoft OCS or Lotus Sametime. Since XMPP has more open source libraries for coding against it, we decided to go the XMPP route. Personally I was happy since the peer to peer nature of the SIP protocol introduces a few problems on our network.
Of course, deciding on XMPP doesn't really narrow things down too much. There are a number of open source XMPP/Jabber servers out there, and quite a few commercial ones as well.
We looked at Jabberd2, eJabberd and Jabber XCP.
As far as I can tell, Jabberd2 and Jabber XCP are very similar in codebase, at least judging by the configuration file format. The thing that the commercial Jabber XCP product provides is a nice web interface for configuring the service, a pair of supported clients (one Windows client, one web based client) as well as commercial support.
eJabberd was impressive. It is an open source application written in Erlang, which I wasn't familiar with. Erlang was a programming language created for distributed computing by Ericsson. On smaller installations, it can handle clustering without any form of external database. It automatically manages synchronization between nodes of the cluster.
Version 2.0 of ejabberd also included a very nice web interface for managing the service, and included an inpressive number of plugins. Configuring the service was very very easy.
At the end of the day, we ended up going with Jabber XCP. The main reason is the one feature that no one else was able to provide. The ability to really tie into the public IM systems, or at least one of them. Any XMPP server can tie into other XMPP services (like Google Talk), but Jabber XCP offered the ability to tie into AIM as well.
The majority of Jabber/XMPP servers offer transports for the various IM services. What these transports do is essentially allow you to log into your own AOL/MSN/etc account from within your IM client. So I may be myname@company.com within my company, but I would be logging into my gdfuego account on AIM.
Jabber XCP has a plugin (for additional per user cost), which allows you to actually use the same myname@company.com address within AIM itself. Unfortuantely MSN and YIM aren't an option at this time.
Right now I'm getting close to turning our new server live, and I've been hitting a number of snags, but that'll be a story for another day.
Saturday, May 03, 2008
Even less spam
Due to my frustration with my personal e-mail, I recently implemented some additional spam filtering which was amazingly effective.
In the past 24 hours, the system has stopped 775 spams destined to valid users, and has accepted 37 legitimate messages. Previously it would have accepted those messages. Most would end up in my spam folder to be reviewed later, but others would end up in my inbox.
This decrease in spam hitting my inbox is through Greylisting, implemented using Postgrey.
A properly implemented mail server needs to be able to deal with a temporary delivery failures, and try again later. A system implementing Greylisting will keep track of the source IP address as well as the source and destination e-mail addresses. The first time a unique combination of these three things is seen, the message is given a temporary failure for 10 minutes.
Legitimate mail is eventually delivered. Spammers give up and don't come back.
In the past 24 hours, the system has stopped 775 spams destined to valid users, and has accepted 37 legitimate messages. Previously it would have accepted those messages. Most would end up in my spam folder to be reviewed later, but others would end up in my inbox.
This decrease in spam hitting my inbox is through Greylisting, implemented using Postgrey.
A properly implemented mail server needs to be able to deal with a temporary delivery failures, and try again later. A system implementing Greylisting will keep track of the source IP address as well as the source and destination e-mail addresses. The first time a unique combination of these three things is seen, the message is given a temporary failure for 10 minutes.
Legitimate mail is eventually delivered. Spammers give up and don't come back.
Friday, April 11, 2008
I reject your e-mail
I've always heard that the correct behavior of a mail server is to reject e-mail for local undeliverable addresses rather than accepting them and then bouncing. I never put too much thought into it though until recently when I took over management of our e-mail infrastructure.
When the systems were handed to me, their queues generally had about 800-1000 e-mails
to be delivered at any given point. As I dug into why, I found that the majority of those e-mails were outgoing bounce messages which were undeliverable for one reason or another.
After a few changes, those systems are now rejecting around 1,000,000 spam messages a day. These are messages that would previously have been accepted and sent back out on the internet as backscatter.
Rejecting instead of bouncing allowed me to significantly cut down on the amount of processing power, bandwith and disk space used on these systems. Not to mention cutting down on the amount of e-mail that the backscatter victims were receiving.
Now if only everyone else who runs mail servers would figure this out.
When the systems were handed to me, their queues generally had about 800-1000 e-mails
to be delivered at any given point. As I dug into why, I found that the majority of those e-mails were outgoing bounce messages which were undeliverable for one reason or another.
After a few changes, those systems are now rejecting around 1,000,000 spam messages a day. These are messages that would previously have been accepted and sent back out on the internet as backscatter.
Rejecting instead of bouncing allowed me to significantly cut down on the amount of processing power, bandwith and disk space used on these systems. Not to mention cutting down on the amount of e-mail that the backscatter victims were receiving.
Now if only everyone else who runs mail servers would figure this out.
Monday, April 07, 2008
I read your e-mail
Well, a few more months go by and I find myself responsible for the e-mail system of a 1000+ employee multi-national company. It only took me about a month or so before I had a better understanding of how mailflow works within the company than anyone had in several years.
Its amazing how much crap can build up in a system that passes hands a few dozen employees who never quite put in the effort to fully understand or attempt any sort of cleanup.
Its amazing how much crap can build up in a system that passes hands a few dozen employees who never quite put in the effort to fully understand or attempt any sort of cleanup.
Friday, December 14, 2007
Wow, I suck
Two posts and I vanish for over a year. Wow I suck.
Well, since my last post I've decided to leave my role as a Security Engineer. Instead I'm moving back to a Systems Administration role supporting Unix servers. Security still interests me, but at the end of the day what I enjoy is using technology to solve problems for people. In my security role I found myself finding problems without being able to assist in solving them.
So as of January 2nd I get to go back to my roots, but with a greater understanding of security. Should be interesting.
Well, since my last post I've decided to leave my role as a Security Engineer. Instead I'm moving back to a Systems Administration role supporting Unix servers. Security still interests me, but at the end of the day what I enjoy is using technology to solve problems for people. In my security role I found myself finding problems without being able to assist in solving them.
So as of January 2nd I get to go back to my roots, but with a greater understanding of security. Should be interesting.
Friday, June 30, 2006
SSH Security
I run a linux server that uses SSH as an authentication method. And my server is under attack. There has been over 1000 login attempts from a handful of systems in the past 2 days alone. Since June 1st, 19 different systems have attacked my system over 10,000 times. Chances are, if you're running a system that allows SSH authentication, your system is under attack as well.
For the past few years there has been a remarkable increase in the number of SSH brute force attacks. There are automated scripts out there that scan networks for SSH daemons on port 22, and when found, attempt to log into them using dictionaries of common usernames and passwords.
If you look in your logs, you'll likely see attempts that look something like this:
Jun 30 09:51:58 localhost sshd[32357]: Failed password for invalid user mark from 211.171.202.87 port 41435 ssh2
Jun 30 09:52:02 localhost sshd[32359]: Failed password for invalid user tomas from 211.171.202.87 port 42307 ssh2
Jun 30 09:52:06 localhost sshd[32388]: Failed password for invalid user rpm from 211.171.202.87 port 43192 ssh2
Jun 30 09:52:10 localhost sshd[32390]: Failed password for invalid user jean from 211.171.202.87 port 44095 ssh2
If one of these common username/password pairs actually work against your system, the attacker will gain access to your machine, and likely will use your computer to continue their search for more systems to attack. Your system may also be used as a jumping point for other types of attacks. Your data may be stolen. And who knows what will be traced back to you.
So is there anything you can do to protect yourself? Sure there is. You have a number of options, with different levels of effectiveness.
1) Switch to key based authentication. If you generate an SSH key and use it to provide all access to your system then you're effectively immune against these sorts of attacks. Brute forcing a password using SSH may take days or weeks. Attempting to brute force an RSA based SSH key would take years. Note that you need to REQUIRE keys for authentication, not just use keys. As long as your server still accepts passwords, you're still vulnerable. Regardless of what your standard method for logging in is.
2) If key based authentication isn't an option for you, your next best option is to use a strong password. The longer your password is, and the more random it is, the less likely a password based attack will work. In this case, it's important to ensure that ALL users of the system have strong passwords. Consider using an application like John the Ripper to test the strength of passwords if you have a number of users on the system.
3) An additional option is to move your SSH daemon to a different tcp port. By default SSH listens on port 22. A daemon listening on a different port would be harder to find. Note that unless you mix this will strong passwords you're depending on Security through Obscurity. Obscuring a system like this isn't a bad thing, but it is not a fix all. Assume that someone will eventually find the daemon, and the attack will continue.
4) Restrict who can talk to your SSH daemon. If you're lucky enough to only have a few sources of logins, you can use either application level IP restrictions or firewall rules to ensure you limit who can attempt to login. This works wonders if you trust the people on the allowed hosts list.
The final item I wanted to mention isn't so much a prevention method as a community good deed. Most IPs you see in your logs represent a system that has been compromised by a similiar attack. In most cases, the owner of the system is not aware of what has happened. Or if they are aware, their ISP might be interested to find out about their activities.
If you have some spare time, try tracking down an attacking system. Use tools like dig and whois to attempt to track down the owner of the system and the hosting ISP, and notify them about the attack. Include the log files showing their attack, and explain the issue in plain language. Treat them as a victim, and talk to them in a non-accusitory way.
In a best case situation you'll get a response from the person and they will work to solve the problem. In the worst case scenario you won't get any response. Maybe they solved the problem without saying anything. Maybe they're ignoring you. In either case, at least you tried.
For the past few years there has been a remarkable increase in the number of SSH brute force attacks. There are automated scripts out there that scan networks for SSH daemons on port 22, and when found, attempt to log into them using dictionaries of common usernames and passwords.
If you look in your logs, you'll likely see attempts that look something like this:
Jun 30 09:51:58 localhost sshd[32357]: Failed password for invalid user mark from 211.171.202.87 port 41435 ssh2
Jun 30 09:52:02 localhost sshd[32359]: Failed password for invalid user tomas from 211.171.202.87 port 42307 ssh2
Jun 30 09:52:06 localhost sshd[32388]: Failed password for invalid user rpm from 211.171.202.87 port 43192 ssh2
Jun 30 09:52:10 localhost sshd[32390]: Failed password for invalid user jean from 211.171.202.87 port 44095 ssh2
If one of these common username/password pairs actually work against your system, the attacker will gain access to your machine, and likely will use your computer to continue their search for more systems to attack. Your system may also be used as a jumping point for other types of attacks. Your data may be stolen. And who knows what will be traced back to you.
So is there anything you can do to protect yourself? Sure there is. You have a number of options, with different levels of effectiveness.
1) Switch to key based authentication. If you generate an SSH key and use it to provide all access to your system then you're effectively immune against these sorts of attacks. Brute forcing a password using SSH may take days or weeks. Attempting to brute force an RSA based SSH key would take years. Note that you need to REQUIRE keys for authentication, not just use keys. As long as your server still accepts passwords, you're still vulnerable. Regardless of what your standard method for logging in is.
2) If key based authentication isn't an option for you, your next best option is to use a strong password. The longer your password is, and the more random it is, the less likely a password based attack will work. In this case, it's important to ensure that ALL users of the system have strong passwords. Consider using an application like John the Ripper to test the strength of passwords if you have a number of users on the system.
3) An additional option is to move your SSH daemon to a different tcp port. By default SSH listens on port 22. A daemon listening on a different port would be harder to find. Note that unless you mix this will strong passwords you're depending on Security through Obscurity. Obscuring a system like this isn't a bad thing, but it is not a fix all. Assume that someone will eventually find the daemon, and the attack will continue.
4) Restrict who can talk to your SSH daemon. If you're lucky enough to only have a few sources of logins, you can use either application level IP restrictions or firewall rules to ensure you limit who can attempt to login. This works wonders if you trust the people on the allowed hosts list.
The final item I wanted to mention isn't so much a prevention method as a community good deed. Most IPs you see in your logs represent a system that has been compromised by a similiar attack. In most cases, the owner of the system is not aware of what has happened. Or if they are aware, their ISP might be interested to find out about their activities.
If you have some spare time, try tracking down an attacking system. Use tools like dig and whois to attempt to track down the owner of the system and the hosting ISP, and notify them about the attack. Include the log files showing their attack, and explain the issue in plain language. Treat them as a victim, and talk to them in a non-accusitory way.
In a best case situation you'll get a response from the person and they will work to solve the problem. In the worst case scenario you won't get any response. Maybe they solved the problem without saying anything. Maybe they're ignoring you. In either case, at least you tried.
Wednesday, June 28, 2006
An Awkward Start
Well folks, I'm here again on my umpteenth attempt to start a regular blog of some sort. These attempts go back about 6 years ago, and generally ends with my anti-social behavior taking over once again.
A little background on myself. I've spent the last 6-7 years working as a Linux sysadmin for a large company based in the Boston area. Last year I made the transitition from Sysadmin work with Infosec (Information Security).
For those of you who may not be familiar with Information Security, it is a rather large field centered around defending computer systems. And of course in order to protect a system you also need to be familiar with how to attack a system. As they say, know your enemy.
This blog is going to cover some of my life, and some of my work. Unless you're interested in computer security, you'll probably want to move on and read something else more up your alley.
A little background on myself. I've spent the last 6-7 years working as a Linux sysadmin for a large company based in the Boston area. Last year I made the transitition from Sysadmin work with Infosec (Information Security).
For those of you who may not be familiar with Information Security, it is a rather large field centered around defending computer systems. And of course in order to protect a system you also need to be familiar with how to attack a system. As they say, know your enemy.
This blog is going to cover some of my life, and some of my work. Unless you're interested in computer security, you'll probably want to move on and read something else more up your alley.
Subscribe to:
Posts (Atom)