Automatic Memory Ballooning with MOM

In this post, I will describe how to use Memory Overcommitment Manager (MOM) to automatically manage memory ballooning on a KVM host.

Memory ballooning is a virtualization feature that allows youto apply a “soft” memory limit to your virtual machines.  For example, you can start a virtual machine with 2GB of assigned memory.  If, at a later time, you want to limit this virtual machine to 1GB you could use libvirt (virsh setmem <domain>1048576) or the qemu monitor (qemu> balloon 1024) to activate the memory balloon.

Memory ballooning can also be used if you want to over-commit memory.  This means that you assign more memory to virtual machines than you have available.  Memory overcommitment depends on an assumption that all virtual machines do not need 100% of their memory at all times.  Unused memory can be shifted around based on the actual memory requirements of each virtual machine by dynamically adjusting the memory balloons of each guest.  Obviously we need some automation here.  One emerging solution for KVM/libvirt hosts is MOM (Memory Overcommitment Manager).  MOM is a Python program that can (among other things) dynamically manage memory ballooning according to a user-defined policy.  Many more details about MOM can be found in the wiki.

How to set up MOM

At the time of this writing, the recommended configuration requires the MOM daemon running on the host to connect to a small daemon running in each guest.  As you will see, this is a bit tricky to set up.  Rest assured that we are working diligently to make it easier in the future!

The first step is to download MOM to your host computer.  You may get the latest version from git or download a release tarball.

Install MOM on the host:

  • Change to the directory containing the MOM distribution and install it python setup.py install.
  • Copy the sample configuration file /usr/share/doc/mom/examples/mom-balloon.conf to /etc/mom.conf.
  • Copy the ballooning policy file /usr/share/doc/mom/examples/balloon.rules to /etc/mom.rules.

Perform the following steps to install the mom guest daemon in each virtual machine:

  • Install MOM in the same manner as for the host:  python setup.py install
  • Copy mom-guestd to /usr/local/bin.
  • Arrange for the daemon to be started at boot: /usr/local/bin/mom-guestd & in /etc/rc.local

Configure MOM network access: The MOM host daemon talks to each guest daemon over the network to gather statistics about guest memory usage.  These statistics help MOM to make better memory ballooning decisions.  In order to connect to the guests, MOM relies on a helper script to determine each guest’s IP address.  A sample name-to-ip helper script is provided in the doc/ directory of the MOM source code.   Follow these steps to configure MOM guest network connectivity:

  • Copy doc/name-to-ip to /usr/local/bin/name-to-ip
  • Edit the name-to-ip helper script to suit your environment.  MOM will pass the script a libvirt domain name and expect the script to print an IP address.
  • Modify /etc/mom.conf
    • Set name-to-ip-helper to /usr/local/bin/name-to-ip
    • Add GuestNetworkDaemon to the [guest]->collectors list

Putting it all together

Now that you have MOM and the guests set up you should be ready to start.

  • Start MOM: momd -c /etc/mom.conf -r /etc/mom.rules
  • Start guests

Pay attention to the MOM output.  If everything is working you should see messages like ‘Guest <name> is ready’.  This means that MOM is communicating with the guests and is ready to manage memory ballooning.  If you do not see the ‘ready’ messages, double-check your mom.conf and make sure you added GuestNetworkDaemon to the guest collectors and that name-to-ip points to an executable script that returns the proper IP addresses of your guests.  Also make sure that mom-guestd is configured to start automatically when your guests boot.

The MOM Ballooning Policy

If you’ve gotten this far, you’re probably wondering about the memory ballooning policy that MOM uses by default.  The principle goal of the default policy is to allow the guests as much memory as possible without adversely affecting the host.  MOM watches the host and guests for signs of memory pressure.  As long as host memory pressure is low, the guests will not be ballooned.  When the amount of claimable memory on the host drops below 20% of total RAM, then MOM will start proactively ballooning guests.  The amount of pressure applied to the guests is proportional to the amount of memory pressure seen in the host.  Guests that are not using as much memory are ballooned more aggressively.  When host memory pressure subsides, MOM restores the guests to their full memory allocation.

MOM is designed to be flexible.  If you would prefer a different auto-ballooning algorithm on your host, feel free to edit the balloon.rules file to suit your needs.  For a start, try modifying the variables at the top.  For example, to start ballooning when there is less host memory pressure, increase pressure_threshold,  To alter the balloon change rate, modify max_balloon_change_percent.

Hopefully this tutorial has helped you to set up an auto-balloon environment with KVM and libvirt virtualization.  I look forward to hearing how it works out for you.  We are working hard to better integrate this technology into the core tools and make it easier to use in the future.

Advertisements

About aglitke

I am a software engineer working on Linux, open source software, and virtualization. I am proud to work at Red Hat on oVirt and Red Hat Virtualization with a focus on software defined storage. Other notable projects I have been involved in include: The Linux ppc64 architecture, Linux kernel crash dumps (kdump), Linux huge pages and libhugetlbfs, qemu, libvirt, and the Memory Overcommitment Manager.
This entry was posted in libvirt, MOM. Bookmark the permalink.

24 Responses to Automatic Memory Ballooning with MOM

  1. M. Kulemin says:

    Thank you for your article. I have a question about balloon mechanism in qemu.
    You made a patch for more implicit “info balloon” output with guest memory statistics. (http://www.mail-archive.com/qemu-devel@nongnu.org/msg17843.html). In Fedora 14 (with qemu 0.13 on host and 2.6.32 kernel on Fedora guests) it does not work, but in code of new libvirt version (at least 0.8.7) I saw parsers for memory statistics from “info balloon”. So did this patch add to last qemu (0.14) or I need use another methods to use memory stats.

    I tried to find more information using Google but could not find anything helpful.

    • aglitke says:

      Yes, good question. In order to make guest memory statistics easily available to a host balloon manager, I did add a statistics interface to the virtio balloon driver. In order to work, support is needed from the guest balloon driver, qemu, and libvirt. The code has been accepted upstream in all three places but was later disabled in qemu due to some problems with asynchronous command support in qemu itself. I am working with Anthony Liguori (the qemu maintainer) to get the extended info balloon enabled again. Until then, you’ll need to make use of the network daemon method described in this post.

      • M. Kulemin says:

        The other way to establish connection between guest and host is virtioserial interface (with pipes or serial) – this technique works without network connections.

  2. carlopmart says:

    Fantastic tool, really. I am doing some tests with a RHEL6 host and some are positives and others negatives. For example. Starting momd, log shows me:

    2011-04-23 20:13:15,433 – mom.HostMonitor – INFO – Host Monitor starting
    2011-04-23 20:13:15,439 – mom.GuestManager – INFO – Guest Manager starting
    2011-04-23 20:13:15,468 – mom.HostMonitor – INFO – HostMonitor is ready
    2011-04-23 20:13:15,505 – mom.PolicyEngine – INFO – Policy Engine starting
    2011-04-23 20:13:15,509 – mom.Monitor – INFO – GuestMonitor-rhelfwmgmt starting
    2011-04-23 20:13:15,516 – mom.Monitor – INFO – GuestMonitor-rhelfwmgmt is ready
    2011-04-23 20:13:15,528 – mom.GuestMonitor – WARNING – Output from name-to-ip-helper win2k8admsrv is not an IP address. (output = ”)
    2011-04-23 20:13:15,530 – mom.Monitor – INFO – GuestMonitor-win2k8admsrv starting
    2011-04-23 20:13:15,535 – mom.Monitor – WARNING – GuestMonitor-win2k8admsrv: Collection error: No IP address for guest win2k8admsrv
    2011-04-23 20:13:15,551 – mom.Monitor – INFO – GuestMonitor-rhelclunode01 starting
    2011-04-23 20:13:15,554 – mom.Monitor – WARNING – GuestMonitor-rhelclunode01: Collection error: Network connection to rhelclunode01 failed: [Errno 111] Connection refused

    After some seconds, momd increase memory for one guest:

    2011-04-23 20:13:25,549 – mom.Controllers.Balloon – INFO – Ballooning guest:8 from 1672852 to 1756494
    2011-04-23 20:13:35,589 – mom.Controllers.Balloon – INFO – Ballooning guest:8 from 1756496 to 1835008

    This is ok, but after that momd process dies … Why?? And another question: can I disable momd checks for certain hosts like windows (in my case, win2k8admsrv)?

    Many thanks Adam.

    • carlopmart says:

      Sorry Adam for disturb. But I have a couple of questions after doing more tests. Can I send you my questions privately or can I post here??

    • aglitke says:

      Hmm, MOM should not be crashing ever. Check dmesg to see if the kernel killed it due to OOM contitions. Otherwise, try increasing MOMs logging verbosity to see if anything interesting is logged.

      As for your second question, you can modify the policy to accomplish this. I’ll admit that the following approach is not the most straightforward.

      Add:

      (if (== guest.GetProp("name") "win2k8admsrv")
      (set balloon_size guest.libvirt_curmem)
      0)

      just before line 57 of the balloon rules (in shrink_guest before guest.Control is called). This will undo any change to the balloon value for a guest if its name is win2k8admsrv.

      • carlopmart says:

        Not crash, but it dies. I see why (or almost I think why). Problem is with main-loop-interval option. If I increase this option until 1 hour, momd stops at one hour. If I increase until 2 hours, momd stops at 2 hours …

        Anoher problem is how mom manage guest memory. For example: I have installed a rhel6 guest with 1024MB for currentMemory and with 1792MB with maximum memory. Guest shows with free command, that it only use 384MB but after some seconds mom increase guest memory until maximum param, 1792MB and guest only use 384MB.. Why??

        And another question: why mom doesn’t deallocate guest memory when guest doesn’t use all memory??

        Thnaks Adam.

      • carlopmart says:

        Any help Adam?

      • aglitke says:

        Did you collect the information I requested in my previous comment?

  3. Greg M. says:

    I noticed there are two policy rules files provided – balloon.rules and ksm.rules. The command line for momd only supports specifying one policy rules file at a time. How does one manage both ballooning and ksm at the same time?

  4. andrew says:

    Cool tool, Adam. but how can I use momd to monitor many guests via ip at the same time ? edit the name-to-ip file?

    • aglitke says:

      Yes. Since there is no standardized way to determine a guest’s IP address, you must use name-to-ip to create a mapping between guest name and guest IP for mom. The better solution is to use a guest agent. Mom knows how to collect from qemu-ga (which is in upstream qemu and is also supported by libvirt).

  5. FB says:

    Thank you for sharing this. I’m trying on a test environment (KVM/libvirt), but when I start the momd, it successfully connects to the VM:
    mom.Monitor – INFO – GuestMonitor-vm-test-03 is ready
    but then it it says:
    mom.Policy – ERROR – Policy error: undefined symbol guest.balloon_cur
    any hints ?

    thanks!

    • aglitke says:

      Hi. You are probably missing the GuestBalloon collector in your mom.conf file. Make sure the setting:

      [guest]
      collectors:

      contains GuestBalloon

      We should probably patch the sample config files to include it by default.

      • FB says:

        yes! now it works
        Thanks a lot 😉

      • Richard says:

        Same issue, same fix, no result.
        I’m almost suspecting that the current release of mom-guestd doesn’t read the config file at all. Or am I wrong? (I’m not a python adept unfortunately)

  6. cho says:

    i am getting this error
    2012-12-14 11:53:00,254 – mom – INFO – MOM starting
    2012-12-14 11:53:00,255 – mom.HostMonitor – INFO – Host Monitor starting
    2012-12-14 11:53:00,255 – mom – INFO – hypervisor interface libvirt
    2012-12-14 11:53:00,266 – mom.HostMonitor – INFO – HostMonitor is ready
    2012-12-14 11:53:00,270 – mom.GuestManager – INFO – Guest Manager starting
    2012-12-14 11:53:00,318 – mom.PolicyEngine – INFO – Policy Engine starting
    2012-12-14 11:53:00,319 – mom.RPCServer – INFO – RPC Server is disabled
    2012-12-14 11:53:00,327 – mom.Monitor – INFO – GuestMonitor-fedora13 starting
    /usr/lib/python2.6/site-packages/mom/Collectors/GuestMemory.py:47: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
    self.stats_error(‘getVmMemoryStats() error: %s’ % e.message)
    2012-12-14 11:53:00,328 – mom.Collectors.GuestMemory – WARNING – getVmMemoryStats() error: libvirt memoryStats() is not active

  7. adhi says:

    2012-12-17 09:11:17,753 – mom – INFO – MOM starting
    2012-12-17 09:11:17,754 – mom – INFO – hypervisor interface libvirt
    2012-12-17 09:11:17,754 – mom.HostMonitor – INFO – Host Monitor starting
    2012-12-17 09:11:17,765 – mom.HostMonitor – INFO – HostMonitor is ready
    2012-12-17 09:11:17,769 – mom.GuestManager – INFO – Guest Manager starting
    2012-12-17 09:11:17,810 – mom.PolicyEngine – INFO – Policy Engine starting
    2012-12-17 09:11:17,811 – mom.RPCServer – INFO – RPC Server is disabled
    2012-12-17 09:11:17,816 – mom.Monitor – INFO – GuestMonitor-fedora13 starting
    /usr/lib/python2.6/site-packages/mom/Collectors/GuestMemory.py:47: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
    self.stats_error(‘getVmMemoryStats() error: %s’ % e.message)
    2012-12-17 09:11:17,818 – mom.Collectors.GuestMemory – WARNING – getVmMemoryStats() error: libvirt memoryStats() is not active

  8. Henrik Uggla says:

    Hi!
    I’m trying to get this to work on Ubuntu 12.04 64bit but I get:

    2013-05-21 14:38:07,045 – mom.Collectors.GuestMemory – WARNING – getVmMemoryStats() error: libvirt memoryStats() is not active
    2013-05-21 14:38:12,032 – mom.Monitor – WARNING – GuestMonitor-skgis2: Collection error: Network connection to skgis2 failed: timed out
    2013-05-21 14:38:12,032 – mom.Monitor – DEBUG – GuestMonitor-skgis2: Incomplete data: missing set([‘swap_out’, ‘major_fault’, ‘swap_in’, ‘mem_unused’, ‘minor_fault’, ‘mem_available’])

    Any ideas?

    Regards
    /Henrik

  9. Henrik Uggla says:

    Is there a similar tool for dynamic cpu allocation? Or would that be pointless?

    /H

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s